Computer Vision and Generative AI: Multimodal Use Cases and Risks
In a 2024 AI Connect webinar sponsored by HPE, leaders from HPE, LTK and the University of Texas Medical Branch discussed how generative AI is reshaping computer vision. They covered multimodal RAG, synthetic data, production lessons, governance and data challenges.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Generative AI helps beyond image creation. Cundiff said it can flag mislabeled data and suggest gaps in training sets.
- Multimodal changes RAG. Lacey said retrieval systems must interpret tables and graphs alongside text.
- Good enough can ship. A system that measurably cuts downtime can go to production without perfect accuracy, the panel said; customer-facing chatbots need a higher bar.
- Privacy tests come first. Guo described testing whether a model reveals confidential data depending on how it is prompted.
- Data remains the hard part. Lacey said top AI challenges are "data and then data and then data again."
In a 2024 AI Connect webinar, "Beyond Sight: Exploring the Frontier of Computer Vision with Generative AI Innovations," three practitioners discussed where generative AI is changing computer vision and what slows it down in the enterprise. The session was part of the Generative AI for Data Executives Webinar Series and was sponsored by Hewlett Packard Enterprise (HPE). Brian Mink, co-founder and president of AI Connect, moderated.
The panelists were Kurt Lacey, AI Solutions Business Manager at HPE; Ming Guo, Director of Business Intelligence at the University of Texas Medical Branch (UTMB); and Morgan Cundiff, Data Science and Machine Learning Manager at LTK.
What is computer vision, and how is generative AI changing it?
Lacey defined computer vision as the ability of AI to interpret visual information, whether images, video or, more recently, multimodal models that combine visuals and text. Many of the early successes of AI, he said, came from image processing.
Cundiff said the generative part of generative AI is creating and editing images and turning images into video. Cundiff also saw a quieter benefit: using an LLM or other generative model to check which items in a dataset may be mislabeled, where gaps exist and how to augment data for a more well-rounded model.
Guo pointed to synthetic imaging in healthcare. For rare diseases where clinical trial recruitment is hard, Guo said, generative AI can produce synthetic patient data, including a digital twin of a patient, to broaden training data and make trials more efficient.
What does multimodal mean for RAG?
Lacey explained that multimodal architectures marry computer vision with natural language processing, so teams no longer have to choose between text and images. That matters for retrieval-augmented generation (RAG), he said, because company documents often contain tables and graphs that need to be interpreted alongside the text.
He also said open-source image models from Hugging Face and others are approaching closed models in accuracy, and more customers are feeding their own images to a common open model. Guo added that open models let a company customize the model with its own data.
Where has generative AI reached production?
Lacey described manufacturing quality control. He cited a common statistic that 20% to 30% of manual inspections miss potential failed components, which he linked to a 70% increase in unplanned downtime. A vendor application pre-trained on a type of equipment, he said, helped customers achieve higher uptime.
Mink drew a lesson from that example: perfect accuracy isn't always necessary. If a system measurably cuts downtime and saves money, it can go to production. A customer-facing chatbot that misstates a refund policy is a different case, he said, and raises the question of whether it should be customer-facing at all.
How do democratization and transparency fit in?
Cundiff said generative AI can explain a chart or dashboard image to a non-technical user through back-and-forth questions. The risk, she said, is hallucination, especially when people treat the output as a source of truth.
Guo also described feature visualization, which highlights the parts of an image a model used in a classification, as one route to transparency. Lacey added that no-code and low-code services such as Google Vision AI and Azure's cognitive services lower the barrier to entry.
What challenges do organizations face?
Guo put data privacy and security first, particularly in healthcare. Guo described UTMB testing whether a model connected to enterprise data could return a doctor's demographic data without disclosing confidential details. The model behaved differently depending on how the prompt was phrased, Guo said, and that had to be resolved before production.
Cundiff weighed cost and risk against value, saying "a lot of times the answer is no still" for some use cases. Cundiff also raised ethical questions about training data from authors and artists, and said teams should be intentional about which data a model can access.
Lacey said the top challenges with AI "are data and then data and then data again," citing multi-month projects just to make data accessible and high quality. Mink agreed: "the best AI strategy is a good data strategy."
On governance, Lacey recommended keeping data where it is governed best, often in the company's own data center, and mentioned guardrails from Nvidia. Guo said that even with clear data definitions, the team keeps testing, because there is no established method yet.
What comes next?
Over three to five years, Lacey expected more synthetic data generation, a move from 2D to 3D, and better open-source models. Cundiff predicted faster, smaller models for near-real-time vision, more attention to source data and bias, specialized models, and more modalities such as audio and video. Guo expected domain-specific medical models, built through collaboration between institutions.
Hear from practitioners in person
Data and AI leaders working on these questions will gather at COLLIDE 2027 on Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. AI Connect also has 24 free live webinars on the 2027 schedule; see the webinar page. Registration is coming soon.
Frequently asked questions
How is generative AI used in computer vision?
Panelists cited image generation and editing, synthetic patient data for clinical trials, dataset validation and augmentation, and explaining charts and images to non-technical users.
What is multimodal AI?
Lacey described it as combining computer vision with natural language processing so models can go from text to images and back.
Do computer vision systems need to be 100% accurate?
Not always. The panel said a system that measurably reduces unplanned downtime can justify production use, while customer-facing chatbots call for more caution.
What are the main governance challenges?
Data privacy and security, hallucinations, cost, data access and bias in training data, according to the panelists.
This article is based on remarks by Kurt Lacey, AI Solutions Business Manager, Hewlett Packard Enterprise; Ming Guo, Director of Business Intelligence, University of Texas Medical Branch; Morgan Cundiff, Data Science and Machine Learning Manager, LTK; moderator Brian Mink, Co-Founder and President, AI Connect at AI Connect webinar, Generative AI for Data Executives Webinar Series (2024) on October 17, 2024. More coverage: Webinars.
Filed under: AI Use Cases & Applications, AI Governance & Trust, Data & AI Infrastructure
