Generative AI Meets Responsible AI: Risks and Observability
At COLLIDE 2023 in Atlanta, Fiddler AI co-founder and CEO Krishna Gade outlined the risks of large language models, including privacy leaks, hallucinations, unreliable self-explanations and bias. He argued that AI observability is the missing link for deploying generative AI responsibly.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- AI is not deterministic. Model performance can drift weeks or months after training, creating production risk.
- LLMs add new risks. Gade named privacy leakage, hallucinations, unreliable self-explanations and bias.
- Pressure comes from both sides. Data and AI leaders are told to move fast while managing risk.
- Observability as the missing link. Gade argued that monitoring outcomes, prompts and responses is how enterprises operationalize generative AI.
- Bridge the gap. Explanations should be shareable with business and compliance leaders, not only data scientists.
At COLLIDE 2023 in Atlanta, Krishna Gade, Co-founder and CEO of Fiddler AI, told attendees that enterprises racing to deploy generative AI need responsible AI practices built in from the start. In a 15-minute talk titled "When Generative AI Meets Responsible AI," Gade laid out the risks he saw in large language models and argued for AI observability, monitoring of models and their prompts and responses, as the way to manage them.
Gade, who previously worked as a data scientist and machine learning engineer at Facebook, spoke from the perspective of a vendor of AI observability tools. His claims about what such tooling can do are his own.
Why did generative AI raise the stakes for responsible AI?
Gade said he had lived through several technology shifts in 20 years, including the dot-com, cloud and mobile transformations, and expected most applications to become AI-based in the coming years. Large language models, he said, let enterprises get started quickly, with use cases such as summarizing legal documents, customer service chatbots, internal documentation tools and conversational shopping experiences.
But AI is "a black box for a lot of us," he said, and it had gone wrong before generative models arrived. He cited biased pricing algorithms, faulty fraud detection and a credit card algorithm accused of bias against women as examples of costly failures in financial services and other industries.
He also described a trade-off for regulators. Regulating too fast could curb development, he said, while enterprises are deciding how quickly to adopt AI and how safely to deploy it.
What makes AI different from traditional software?
Gade said a trained model is not deterministic software. Its performance can change weeks or months later, and that drift creates risk once an application is in production. Teams may not know how a prediction was reached, whether a model is biased against a demographic, whether it leaks personal data or sends unsafe responses, or how sensitive it is to small changes in a prompt.
What are the specific risks of large language models?
Gade walked through four areas for generative AI.
Privacy. He said large language models can memorize training data, and prompt injection attacks could let customers or hackers extract it.
Hallucinations. Because models predict the next word, they can "make up things," Gade said. He described a case in which ChatGPT declared a well-known computer science researcher dead, and warned about a customer-facing chatbot that misstates a rate or a price.
Transparency. He said a model's self-explanations can vary with context, so they cannot be relied on to explain its own output. As an example, he said the answer to whether "Wayne Rooney shot from outside the 18" is plausible can change depending on whether the question frames football or soccer.
Bias. Gade said the web is largely authored by people who speak English and have internet access, so bias is built into models by default. Understanding how data influences outcomes is therefore important, he said.
How can enterprises deploy generative AI responsibly?
Gade said chief data officers, chief analytics officers and AI heads face pressure from the top to implement AI as fast as possible while also managing these risks, because if AI goes wrong, "their heads will roll."
His answer was AI observability. He said the industry has mature data platforms and a wide choice of pre-trained models, but lacks transparency into why a model predicted what it did and an overview of all the models a business runs, with governance around them. That, he said, is "the missing link" in operationalizing generative AI.
Monitoring AI outcomes, prompts and responses, Gade said, helps make sure models work for end users and protect brand and reputation. He described Fiddler's platform as a single view where data scientists and ML engineers track model performance, set alerts, run root cause analysis and generate explanations to share with business and compliance leaders. He said this helps close the gap between practitioners and business people, a gap he recalled from his time at Facebook when colleagues asked why an algorithm showed a story and the team could not say.
Where to continue the conversation
Conversations about trustworthy AI continue at COLLIDE 2027, the largest AI conference in the Southern U.S. with more than 1,000 enterprise data and AI leaders, on Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. See /events/collide/. Registration is coming soon. Free live sessions are also on the 2027 webinar schedule.
Frequently asked questions
What are the main risks of generative AI in the enterprise?
Krishna Gade named privacy leakage through memorized training data, hallucinations, self-explanations that vary with context, and bias inherited from web data.
Why is AI different from traditional software?
Gade said a trained model is not deterministic and its performance can change weeks or months later, so it needs ongoing monitoring.
What is AI observability?
Per Gade, it means monitoring AI outcomes, prompts and responses, and how predictions are made, so models can be governed and trusted.
Was this a sponsored session?
The description does not say; Gade presented from Fiddler AI, an AI observability vendor, and his product claims are his own.
This article is based on remarks by Krishna Gade, Co-founder and CEO, Fiddler AI at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: AI Governance & Trust, Observability & Data Quality, AI Use Cases & Applications
