Data Quality and Observability: Garbage In, Garbage Out Panel (2023)

Data Science Innovators Conference 2023EXPERT PANEL: Garbage In, Garbage Out: Leading Edge Approaches to Dealing with Classic Data ProblemsWatch the full session · 32 min

In a panel at the Data Science Innovators Conference 2023, "Garbage In, Garbage Out: Leading Edge Approaches to Dealing with Classic Data Problems," data practitioners from Snap, Wells Fargo and CVS discussed why data quality fails and how observability can help. Selma Dogic, Senior Manager of the Analytics COE at Carter's, moderated. The panelists were Sarah Nooravi, Revenue Strategy Lead at Snap; Eric Voots, Lead Software Engineer for Cybersecurity and Machine Learning at Wells Fargo; and Shashank Goswami, Senior Data Scientist at CVS.

What causes poor data quality?

Voots said the problem usually starts where data is created. In his experience in finance, there are few controls at the origin, with Excel used as a database or files passed back and forth over email. "It's usually a process problem," he said, and one that has often gone unfixed for years. He gave a simple example: no check stops someone from typing letters into a numeric field.

Nooravi grouped the causes into ownership, communication and data sources. People who notice an issue may not escalate it, escalation channels may not exist, and downstream processes get built on bad data and turn fragile. With many pipelines and no observability, she said, engineers depend on subject matter experts to surface problems. Observability flips that, putting more responsibility on engineers to catch issues before they travel downstream.

Goswami pointed to siloed source systems in large organizations. When those systems are merged into a unified dataset, he said, the result can be messy and unreliable, so standardization at the source matters.

What does bad data cost businesses?

Voots recalled a case in corporate finance in which a data vendor supplied a field for a large public company that shifted from its normal value, with no reasonableness check. It directly affected trades made that day, he said. More commonly, he added, bad data shows up as a dashboard where a normal line suddenly changes and nobody knows why.

Nooravi described quarterly planning at Snap, when CRMs are updated and third-party data is integrated across many teams. That work is resource intensive and can delay reports that drive strategy early in a quarter.

Goswami noted that data can be accurately captured and still unusable if it lacks the relevant fields. He cited early pandemic prediction models, many of which, he said, strayed far from actual numbers, while a simple model built on a few accurate inputs by a data scientist in finance outperformed better-funded efforts.

Can automation and AI help monitor data quality?

Goswami said communication between domain experts and the engineers and architects building the data warehouse leads to data that is intuitive for scientists, "rather than a mystery ball." Voots said his organization had no choice but to move to tools such as recommendation engines and Spark to handle customer volume, because older tools like SAS cannot keep up.

Looking ahead, Goswami suggested large language models could help create data validation rules at source systems and flag inaccuracies in large datasets. Nooravi pictured automated validation covering 80 to 90 percent of the work, with humans checking recommended fixes. Voots said unlabeled security data makes outlier detection and dashboards essential, because "the hackers don't label our data." He also hoped data ethics would get more attention as dashboards improve.

How should leadership and culture support data quality?

Nooravi said it starts with leadership buy-in. Without data engineering support, she said, analysts and data scientists build their own pipelines, which are fragile and often undervalued, and which break when the builder leaves. At Snap, she said, teams have moved toward a centralized hub for shared datasets with named pipeline owners, though no single structure fits every organization.

Voots named transparency as the top thing leaders must enforce, without punishing people for side processes, since a consequence model just teaches people to hide them. In his words, "sunlight it's the best disinfectant." He also said leaders talk about wanting AI but rarely about platform engineering first, which data scientists need.

Dogic noted that governance and data quality teams can be the first cut, and said where a chief data officer reports reveals a company's attitude toward data.

Where can you keep learning about data quality and observability?

Data quality and AI readiness remain central topics for enterprise data leaders. They are on the agenda at COLLIDE 2027 on Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta, which organizers call the largest AI conference in the Southern U.S. Details are at /events/collide/; registration is coming soon.

Frequently asked questions

What causes poor data quality?

Panelists cited missing controls at data entry, unclear ownership and escalation, and siloed source systems merged into messy unified data.

What does poor data quality cost?

Examples included a bad vendor data field affecting trades, broken dashboards, and delayed planning reports.

How can organizations improve data quality?

Leadership buy-in, data engineering support, centralized pipelines with owners, transparency and observability tooling.

Can generative AI help with data quality?

Goswami suggested LLMs could help create validation rules at source systems and flag inaccuracies.

This article is based on remarks by Selma Dogic, Sr. Manager, Analytics COE, Carter's (moderator); Sarah Nooravi, Revenue Strategy Lead, Snap; Eric Voots, Lead Software Engineer - Cybersecurity and Machine Learning, Wells Fargo; Shashank Goswami, Sr Data Scientist, CVS at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.

Filed under: Observability & Data Quality, People, Culture & Leadership