Data Observability for GenAI: Deloitte, Monte Carlo and PRGX Panel
In a 2024 AI Connect webinar, leaders from Deloitte, Monte Carlo and PRGX Global explained why data quality and observability underpin generative AI. They covered the five pillars, how to start, what to look for in tools and common pitfalls.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Five pillars. Monte Carlo's May defined data observability by volume, freshness, schema, lineage and field quality, tied to the code and systems behind the data.
- Trust drives value. Deloitte's Sharma said observability builds trust in data, citing regulatory reporting at banks.
- Detection is not enough. Teams need triage processes that route issues to the team that owns the data.
- Governance favors vendors. Amit said his team moved from open source to a vendor tool because open-source options lacked governance.
- Business owns quality. Panelists said business data owners, not just engineers, must be accountable for data quality.
In a 2024 AI Connect webinar, part of the "Generative AI for Data Executives" series and brought to you by Deloitte and Monte Carlo, three data leaders explained why data observability is a prerequisite for generative AI. The panel included Sharma, a principal and data operations practice leader at Deloitte Consulting LLP, May, a founding product manager at Monte Carlo, and Amit, chief AI officer at PRGX Global. A AI Connect co-founder moderated.
The panelists covered what observability means, how to start, what to look for in a tool and what tends to go wrong.
What is data observability and why does it matter for AI?
May said Monte Carlo defines data observability around five pillars: volume, freshness, schema, lineage and field-level quality. It also correlates the data with the systems and code that shape it, such as pipeline jobs and dbt models, she said.
Sharma described the same idea as three lenses: the data itself, the pipelines that move it and the underlying infrastructure. Bringing them into one view lets operations teams see whether a problem is a data issue, an infrastructure issue or a query that needs tuning, he said. Amit added that data engineers, data scientists and business stakeholders each care about different parts, and data scientists also care about drift over time.
On the AI side, Sharma said trust is the core benefit and gave the example of banks whose regulatory reports cannot be trusted without clear lineage. Amit said generative models' tendency to hallucinate makes observability more important. May said the most relevant generative AI products rely on a company's own data, whether through prompts, fine-tuning or retrieval from vector databases, so that data has to be healthy.
Does AI still need clean data if it can handle messy data?
An audience member asked that directly. Sharma said models trained on internet data can synthesize general information, but an executive deciding where to spend next quarter's budget needs accurate sales data behind the chatbot.
May said teams still tend to convert unstructured data to structured data before running analytics. Amit said the question applies most to unstructured data and large language models, while quality matters even more with structured data and classical machine learning.
How should teams get started with data observability?
May said many teams begin with open-source tests from tools such as dbt or Great Expectations. At scale, with thousands of tables across warehouses, pipelines and BI tools, broad automated coverage becomes necessary and is hard to build in-house, she said. Detection is only the start: "knowing that there's issues that happen with your data is only the first step."
Teams also need a process that routes each incident to the team that owns the data, she said. Sharma looked at it through people, process and technology and said governance is no longer a separate offline function: "everything is now getting embedded into the real time flow of data."
Amit described the path his previous team took: rules-based checks, then open-source libraries, then a vendor tool. The open-source options lacked governance, he said, such as who gets notified and what protocol follows, and building it would have taken years.
What should you look for in an observability tool?
Panelists offered a checklist:
- Data quality. Scalable checks across cloud and hybrid setups, anomaly detection and drift detection, according to Sharma.
- Pipeline and infrastructure telemetry, with wide integrations. Monitoring one warehouse alone does not solve most clients' problems, Sharma said.
- Monitoring, triage, resolution and reporting, which May listed as a vendor's core functions, including table- and field-level lineage to trace an issue to its source.
- Metadata quality, which Sharma said orchestrates jobs and data movement.
- Business-friendly output. Amit said dashboards should be simple enough for business stakeholders, and Sharma noted governance is shifting toward data products owned by business people.
Does data observability explain AI decisions?
Not by itself, the panelists said. May called explainability a separate problem, handled by an emerging category sometimes called ML or generative AI observability. Sharma said lineage still shows where the data feeding a model came from. Amit suggested bringing data and model governance under one umbrella and expected the tools to converge.
What goes wrong, and what comes next?
Sharma named the variety of systems to integrate, treating observability as purely technical, and the rapid growth of unstructured data. May said tying the work to business value is hard for companies without obvious revenue levers like pricing engines. Amit said business owners must take on data quality roles, since treating it as a technical problem has wasted money and time.
Looking ahead, May expected more adoption, traditional data quality vendors moving into observability and warehouses and ETL tools adding native checks. She said the best tool will still be cross-platform and end to end.
Where can you hear more from data and AI leaders?
AI Connect, formerly AI Connect, hosts 24 free live webinars on its 2027 schedule, including the AI × Enterprise Panel Series and Plug & Play Series. Registration is coming soon. See /webinars/.
Frequently asked questions
What are the five pillars of data observability?
According to Monte Carlo's May, they are volume, freshness, schema, lineage and field-level quality.
Does AI still need high-quality data?
The panelists said yes. Enterprise decisions need accurate data behind generative AI, and quality matters even more for structured data and classical machine learning.
Does data observability explain AI model decisions?
Not directly. May called explainability a separate problem, though lineage shows where model input data came from.
What should you look for in a data observability tool?
Panelists listed scalable quality checks with anomaly and drift detection, wide integrations, lineage, triage workflows, metadata quality and business-friendly dashboards.
This article is based on remarks by Sharma, Principal and Data Operations Practice Leader, Deloitte Consulting LLP; May, Product Manager, Monte Carlo; Amit, Chief AI Officer, PRGX Global at AI Connect webinar (Generative AI for Data Executives series) on August 28, 2024. More coverage: Webinars.
Filed under: Observability & Data Quality, AI Strategy & ROI, Data & AI Infrastructure
