RAG Explained: Embeddings, Vector Databases and LLM Choices
At COLLIDE 2023, Redis principal solution architect Anton Umnikov explained how transformers led to embeddings and ChatGPT, and how a RAG system fits together. His advice: pick embeddings, vector database and LLM independently, and remember that vectors are easy while databases are hard.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- One paper, two branches. Umnikov said embedding models and ChatGPT-style models both descend from the 2017 transformer paper.
- RAG needs three parts. Embeddings, a vector database and a language model, each chosen independently.
- Bring your own embeddings. He said embeddings are light enough to run yourself and can be a large share of an API bill.
- Vectors are easy, databases are hard. Security, scalability and availability should drive vector database choice.
- More than generation. Vector search also supports semantic caching and conversation memory for LLM apps.
At COLLIDE 2023 in Atlanta, Anton Umnikov, Principal Solution Architect at Redis, walked attendees from the transformer research that started the generative AI wave to the practical choices involved in building a retrieval augmented generation (RAG) application. In a roughly 22-minute talk, Umnikov covered where embeddings and large language models come from, how the pieces fit together and where he thought teams should spend, or save, their money.
His practical advice, as presented at the time: choose the embedding model, the vector database and the language model independently, and look closely at the database half of "vector database."
Where do embeddings and ChatGPT come from?
Umnikov traced both to the 2017 paper "Attention Is All You Need," which introduced the transformer. The paper described two parts, an encoder and a decoder, which its authors used together for machine translation, first English to German and then English to French.
After publication, he said, people began using the two halves independently. The encoder, trained by masking a random word and predicting it with context from both sides, led to BERT and its derivatives, which extract semantic meaning from text. The decoder predicts the next word from the text to its left, and that, Umnikov said, is the basis of GPT. ChatGPT and Llama 2 are "essentially decoder Transformer from our original paper on steroids."
He described embedding models as the smaller, lightweight cousins of generative models. An encoder has hundreds of millions of parameters and takes up hundreds of megabytes to a few gigabytes, while decoder models are huge and need specialized hardware.
What is retrieval augmented generation and why use it?
Umnikov placed RAG between traditional search such as Google, Bing or SharePoint and purely memory-based models such as ChatGPT. The first practical reason, he said, is that a model's prompt window is limited: "you can't feed all your internal data into ChatGPT," so the relevant data has to be selected first. Other benefits he listed were fewer hallucinations, observability and traceability.
What components does a RAG system need?
He said a deployment needs three components: an embedding generator, a vector database and a large language model. In the pattern he described, documents are stored, split into chunks and loaded into the vector database as embeddings. When a user asks a question, the same embedding engine converts it, the most similar chunks are retrieved and they are passed to the language model.
Umnikov showed reference setups on Azure with Azure OpenAI and on Google Cloud with Vertex AI's PaLM model, and said that on Azure a Terraform apply can stand up an infrastructure private to the organization. There is no dependency, he said, between the embeddings and the language model, and Redis was used as the vector database in his examples, though he said teams are not limited to it. He pointed to the Redis Ventures GitHub for the examples.
He added that vector embeddings have uses beyond generative AI. He described an Australian company, Relevance AI, that used embeddings and Redis vector search to identify cheaters in an online multiplayer game by looking for outliers in gameplay.
Why should teams bring their own embeddings?
Umnikov argued against defaulting to the embeddings from the model vendor. He said Hugging Face maintains a benchmark of embedding models that can run on a laptop or cloud instance, and that OpenAI's Ada ranked tenth on it, not first.
Cost was the bigger point. He said OpenAI's price per token for embeddings was 20 times lower than for generation, but because a team must embed its entire corpus repeatedly while tuning chunk size and overlap, the spend adds up. On one project, he said, "70% of my OpenAI bill was for embeddings." Using a framework such as LangChain, he said, switching to a Hugging Face model takes about two lines of code. He would think twice about provisioning infrastructure to run Llama 2 yourself, he said, but for embeddings it is a "no-brainer."
How should you choose a vector database?
Vector similarity is math, Umnikov said, such as cosine similarity, and it is well known. The hard part is the database: scalability, high availability and especially security in an enterprise setting. His summary was "vectors are easy, databases are hard."
He said some traditional enterprise customers talk to Redis not because they see it as the best vector database but because it is an approved vendor with the needed certifications, which at a large bank can take up to a year to establish. Between pure-play vector databases that still have to catch up on enterprise features and traditional databases adding vector search, he said he would bet on the latter "at this moment of time," while noting that he works for Redis.
He also listed other roles for a database like Redis in LLM systems: context retrieval, a semantic cache for model responses and memory for conversations. He said ChatGPT's March 2023 outage, in which some users briefly saw other customers' conversations, traced to a client-side bug in code that connected to Redis, and that OpenAI continued to use it afterward.
Where to learn more
COLLIDE 2027, billed as the largest AI conference in the Southern U.S., takes place Wednesday, October 13, 2027, at Sandy Springs Performing Arts Center in Atlanta. The AI Enterprise Conference 2027 is Tuesday, August 31, 2027, at Pier Sixty in New York City. Details are at /events/collide/ and /events/ai-enterprise-conference/. Registration is coming soon.
Frequently asked questions
What is retrieval augmented generation?
Umnikov described RAG as an architecture between traditional search and purely memory-based models: relevant text is retrieved from your own data and passed to the language model.
What components does a RAG system need?
An embedding generator, a vector database and a large language model, which he said can be chosen independently.
Why use your own embeddings instead of the LLM vendor's?
He said embeddings can run on your own hardware, can outperform vendor options on public benchmarks and, on one project, accounted for 70% of his OpenAI bill.
How should you choose a vector database?
Umnikov said the vector math is well known, so evaluate the database side: scalability, availability, security and enterprise certifications.
This article is based on remarks by Anton Umnikov, Principal Solution Architect, Redis at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: Data & AI Infrastructure, AI Use Cases & Applications
