From Data to Generative AI: Building On-Premises RAG Apps

AI Connect webinar (2024)Getting From Data to Generative AI QuicklyWatch the full session · 51 min

In a 2024 AI Connect webinar, "Getting From Data to Generative AI Quickly," Tyler Britten, AI Solution Engineer at Hewlett Packard Enterprise (HPE), walked through how a company can go from raw documents and databases to a working generative AI application running inside its own data center. The session was sponsored by HPE and included live demos of model serving, retrieval-augmented generation (RAG) and text-to-SQL.

How do large language models get built, and what do enterprises actually need to do?

Britten described the lifecycle in stages: defining the model's structure, pre-training on large amounts of data using many GPUs, instruction tuning so the model follows requests, optional fine-tuning, and finally inference inside an application. Models whose names end in "instruct," he noted, have already been instruction tuned.

Most enterprises, he said, start well past the expensive early stages. They pick a pre-trained, instruction-tuned open model and may never fine-tune it. Britten said fine-tuning tends to change how a model writes, such as matching contract language, more than it teaches the model new facts.

How can a company run a model inside its own walls?

Britten said he is often asked this by teams that are allowed to use generative AI but not outside services because of security restrictions. In the first demo he used HPE's machine learning inference software to pull a Llama 3.1 8B Instruct model from a public registry, assign it one GPU and serve it. The endpoint was compatible with the OpenAI API, so any tool built for that API could connect, and he pointed an open-source chat interface at it.

A dashboard showed requests per second, successes and logs. The result, he said, is an experience like a commercial chat service, hosted on premises.

What is RAG, and why use it instead of fine-tuning?

A general model knows only what was in its training data. In RAG, documents are split into chunks, converted by an embedding model into vectors and stored in a vector database. A user's question is embedded the same way, the closest chunks are retrieved, and the model is told to answer using them.

"By limiting it to the context we're sending it, it makes it much more accurate," Britten said, compared with fine-tuning a model on company data. Developers can still control the model, temperature, system prompt, guardrails on what may be asked, how many chunks to retrieve (top K) and a similarity cutoff. In the demo he used a 70% cutoff.

The interface also displayed sources. Britten showed an answer from Army aviation documents that linked to the exact page of the source, which he said matters in expert-assist cases where users already know roughly what the answer should be and need to verify it.

How do you get documents into a RAG pipeline?

For ingestion, Britten used HPE's machine learning data management platform, formerly the open-source Pachyderm. It processes documents in versioned pipelines, similar to commits in GitHub, and skips files it has already handled. In the example, seven new PDFs took 19 minutes with two workers because scanned pages require full OCR. More workers could be added.

He used the open-source tool unstructured.io to extract tables and figures, and said teams should tune it to their own data. Chunking matters too. He advised keeping related content together, not splitting a table in half, and keeping chunks small enough to avoid a very large context window. Tables, he said, are best supplied to the model as HTML, because models are heavily trained on it.

Can RAG work on structured data?

Yes, Britten said. In another demo, the model converted a question into a SQL query, ran it against a database and answered from the results. He used Vanna AI, which stores the database schema, example questions and short documentation, such as noting that users who say "records" mean albums, to help the model write better queries.

Asked whether examples must be entered by hand, he said the tool learns as users save corrected queries. Typically one or two similar examples are enough to start, and the additions tail off over time. He said this approach has grown in business intelligence, where users previously had to open a ticket to get a new dashboard.

What does it take to move from demo to production?

Britten said the model itself changes little. Most development happens in the code around it, such as frameworks like LlamaIndex or LangChain, so teams should apply normal application practices: CI/CD, testing and regular evaluation against a set of questions with known answers. Red-team tests should check whether the model can be pushed to leak data. "It's just like traditional CI/CD," he said, with LLMs at the center.

A more advanced demo, an owner's-manual assistant built by a colleague, extracted model names, years and body types as metadata, and filtered the vector search on them. Britten said PGVector is one option, because many customers already know Postgres: "Why also bring a brand new database as well?" He also described HNSW indexes, which speed up search at the cost of slower ingestion.

He added that traditional models can still beat LLMs on cost and accuracy, citing a risk model that scores credit card transactions. He also said agentic setups, in which an LLM decides which tools or sub-agents to run, are the next step, and that many users find the 8B Llama model good enough rather than the 70B or larger.

Learn more at upcoming AI Connect events

Teams building RAG and agent applications can hear from enterprise practitioners at COLLIDE 2027, Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. AI Connect also has 24 free live webinars on its 2027 schedule; see the webinar page for details. Registration is coming soon.

Frequently asked questions

Can a company run generative AI without external services?

Yes. Britten demonstrated serving an open Llama model on premises with an OpenAI-compatible endpoint, so existing tools can connect to it.

Why use RAG instead of fine-tuning?

Britten said limiting a model to retrieved context makes answers more accurate, while fine-tuning is better for changing how a model writes.

Can RAG work with structured databases?

Yes. In his demo the model generated a SQL query, ran it and answered from the results, using Vanna AI to store schema and example queries.

Do generative AI models replace traditional ML models?

Not always. Britten said smaller traditional models, such as credit card risk scoring, can be more effective on cost and accuracy.

This article is based on remarks by Tyler Britten, AI Solution Engineer, HPE at AI Connect webinar (2024) on October 29, 2024. More coverage: Webinars.

Filed under: Data & AI Infrastructure, AI Use Cases & Applications, AI Strategy & ROI