Democratizing LLMs: Open-Source Models, RAG and AutoML at H2O.ai

COLLIDE Data + AI Conference 2023Democratizing Large Language ModelsWatch the full session · 24 min

At COLLIDE 2023 in Atlanta, Sri Ambati, Founder and CEO of H2O.ai, used a 24-minute session to demonstrate open-source large language models and retrieval augmented generation (RAG), then argued that data scientists were about to take on much larger roles inside companies. His claims reflect the view of the founder of a vendor that sells these tools.

Ambati opened by thanking the data science community and saying his company's aim had always been, like water, "to see AI everywhere." He framed the session around using generative AI to tell more powerful stories with data.

How can companies run open-source LLMs and RAG?

Ambati said H2O GPT is fully open source and built on Meta's Llama 2. He said that in a handful of steps, attendees could run a quantized version of Llama 2 on a laptop using the ggml library, though slowly.

In a live demo, he loaded the Llama 2 research paper and asked the model how Llama 2 differs from Llama 1. RAG, he explained, creates embeddings of the document on the fly and moves the system "from a pure LLM mode" into answering from that particular document. He compared a 7-billion-parameter model's answer, which he said was off, with the larger 70-billion-parameter model.

He then showed an enterprise edition with fine-grained access controls. Using a customer slide deck, he asked it to describe ways H2O is helping Commonwealth Bank of Australia, and said the system pulled in sub-slides referenced by the main slide. His point was that with good prompts, companies can deliver strong experiences on internal documents "that don't need to go into ChatGPT" or other public services.

Ambati described the bank as a customer that uses H2O for next-best-conversation applications and has more than a thousand models in production, including for credit scoring.

How do you evaluate and customize LLMs?

Ambati said H2O had built an evaluation tool that pits LLMs against each other using an Elo ranking, similar to chess players, with a top model acting as judge. At the time, he said, the March 14 version of GPT-4 ranked best among models available as a service.

For customization, he showed Data Studio, which converts documents into question-and-answer pairs generated by an LLM that can then be used for fine-tuning. He also demonstrated generating dashboards and apps from a sketch using code-generation models. "This is the time to be a maker," he said, arguing that generative AI can multiply individual impact.

How does AutoML combine with LLMs?

Ambati described what he called a convergence of machine learning and LLMs, in which models use features and Shapley values to drive narratives and decisions.

His main example was AT&T, which he said asked H2O to determine which telephone poles were most likely to be hit by a hurricane. Working with Argonne National Laboratory, the team found through automated feature discovery that precipitation 43 days before had a large effect on when a pole was likely to fail. The model was built with Driverless AI, the company's AutoML platform.

He said the platform runs a genetic algorithm to choose models and features, guards against leakage and overfitting, and generates code from the model. Ambati claimed it gives consistency even when a data scientist leaves, and that it balances accuracy, speed and interpretability. He also cited predictive maintenance, factory-floor analytics and demand sensing as common use cases.

Where does competitive advantage come from in the generative AI era?

Ambati said much of the technology problem has been solved and that culture and leadership are now the hard part. "Courage is still a very scarce resource," he said.

He argued that prompts are intellectual property ("Prompt is your IP," he said) and that a strong community and customer base is the real moat. He also predicted that by 2030 every Fortune 500 company would be led by data scientists or AI engineers. That is his forecast, not a finding.

He pointed to customers using data science for strategy, including Chipotle and Reliance, and an Australian customer weighing options under U.S.-China geopolitical tension. His closing message to students: AI is "a true co-founder for anyone with purpose."

Where to hear more on enterprise AI and open-source models

COLLIDE 2027, billed as the largest AI conference in the Southern U.S. with more than 1,000 enterprise data and AI leaders, takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. Details are on the COLLIDE event page, and registration is coming soon.

Frequently asked questions

What is retrieval augmented generation (RAG)?

Ambati described it as creating embeddings of a document on the fly so the LLM answers from that document rather than only from its training.

Can companies run Llama 2 locally?

He said that using quantization with the ggml library, Llama 2 can run on a laptop in a few steps, though not quickly.

How does H2O.ai evaluate LLMs?

Its evaluation tool runs LLMs against each other with an Elo ranking, using a top LLM as the judge.

What did AT&T use AutoML for in Ambati's example?

To identify which telephone poles were likely to be hit by a hurricane; automated feature discovery found that precipitation 43 days earlier had a large effect.

This article is based on remarks by Sri Ambati, Founder & CEO, H2O.ai at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.

Filed under: AI Use Cases & Applications, AI Strategy & ROI