Scaling RAG Beyond Vectors: Hybrid Search, Evals and Reranking

AI Connect webinar (2024)Scaling AI Beyond Vectors: Building Smarter, Faster Systems for Retrieval-Augmented Generation (RAG)Watch the full session · 54 min

In a 2024 AI Connect expert panel webinar, "Scaling AI Beyond Vectors: Building Smarter, Faster Systems for Retrieval-Augmented Generation (RAG)," two practitioners argued that a vector database alone does not make a production RAG system. Jon Bratseth, CEO and founder of Vespa.ai, and Aakarsh Ramchandani, Chief Strategy Officer at RavenPack, spoke to an audience the organizers described as hundreds of attendees from around the world. A AI Connect co-founder moderated, and the session was sponsored by Vespa.ai.

What is RAG, and where do vector databases fit?

Bratseth described RAG as a three-step pattern: work out which data is relevant to a problem, retrieve it, and hand it to the large language model as part of the prompt. Models don't have a company's private or up-to-date data, so that data has to be supplied.

Vector databases, he said, emerged almost by accident as one way to do the retrieval. Text is split into chunks, each chunk is converted into a vector, and the system retrieves chunks whose vectors sit close to the vector for the question. But "there's no actual connection" between the AI that retrieves and the AI that generates, he noted, so any retrieval method can feed the model, from a relational database to a traditional search engine.

Ramchandani said RavenPack came to RAG from search. In his view, users often want an answer rather than a list of links, and vectors are one powerful feature in a larger retrieval pipeline.

What does it take to scale a RAG system?

Bratseth separated two scaling problems. The first is running the large language model, which he advised outsourcing to providers with the GPUs and economies of scale "unless you're one of them." The second is the data: vectors, text, metadata and extracted entities, which quickly outgrow a single node.

He said that data has to live in one distributed system, because joining vectors, text and metadata across separate systems at query time wastes bandwidth. The smaller machine learning models that rank results should also run where the data is stored, rather than shipping data elsewhere for inference.

Ramchandani framed scale along three lines: content, inference and distribution. At a billion-vector scale, he said, the question is how to store enough vectors "not in memory to bankrupt yourselves." Real-time systems also need freshness signals that decay at different rates, and cross-encoder rerankers that must narrow hundreds of candidates within a latency budget he put at 200 milliseconds to one second.

Why is cosine similarity alone not enough?

Both panelists were blunt. Ramchandani said cosine similarity may work for a few thousand documents, but at large scale embeddings are "a feature that's part of a much larger pipeline." Much of the payoff, he said, comes from cross-encoder reranking.

Bratseth said academic benchmarks show that cosine similarity alone does not give good relevance. If a team wants only one simple thing, he said, traditional BM25 text search is cheaper and often better. The best quality, he said, comes from hybrid search, which combines vector similarity with text search and merges the signals. Text search is precise when you need an exact product code or person, while vectors help when the query is a concept with many possible phrasings.

How do you measure quality before scaling?

Evaluation sets were the panel's most repeated theme. "If you don't have that there's no point in actually doing any scaling work," Ramchandani said. His team kept a static cluster of about 100,000 documents and a set of questions, and tracked NDCG scores whenever a new ranking algorithm was deployed. Later, he said, teams need to sample live traffic too, because real users ask different things.

For company-specific data, he said the easiest start is a deliberately rough search box that people can try, since their questions become the evaluation set. His team also fine-tuned a sentence transformer on domain-specific sentence pairs, and indexed company-specific metadata, such as detected entities, so queries could use it. Off-the-shelf models, he said, are where many client proofs of concept fail.

What about embeddings, chunking and newer retrieval models?

On cost, Ramchandani recommended training a smaller student model from a larger one, which he said loses little quality. His team, which he said receives bursts of about 40,000 documents a second, was also exploring binary and Matryoshka-style embeddings. Bratseth explained that Matryoshka vectors put the most important numbers first, so vectors can be shortened.

On chunking, Ramchandani said RavenPack builds its own chunker because documents range from a headline to 300 pages, and found a sweet spot of up to 512 tokens per chunk.

Both were enthusiastic about ColBERT, a token-level retrieval model that creates many vectors per document. Bratseth said combining it with binary vectors makes it economic at scale. He also pointed to ColPali, which indexes document pages as image patches and skips text extraction. He said that with the new work, it is not more expensive than ColBERT, and quality is higher.

Where is RAG heading?

Bratseth expected more compute to shift from training to inference, so that one question could trigger thousands of queries against enterprise data as a model explores possible answers. That would create a new scaling problem: handling the torrent of queries.

Ramchandani described dividing queries into classes, each with a router and its own workflow. "You don't actually need to go solve everything with RAG," he said. Because "this is a systems design problem," he added, teams should start from what users are trying to achieve.

In the closing lightning round, Bratseth's advice was to start with evals and simple BM25, and to add complexity only as evals prove it helps. "Always prove that you're adding value," he said.

Meet the builders in person

Practitioners working on retrieval and AI infrastructure gather at COLLIDE 2027, Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. See the COLLIDE page for details, or browse the 24 free live sessions on the 2027 webinar schedule. Registration is coming soon.

Frequently asked questions

What is retrieval-augmented generation (RAG)?

Bratseth described it as finding data relevant to a problem, retrieving it and giving it to the large language model in the prompt, since the model lacks private or current data.

Is a vector database enough for RAG?

Not according to the panel. They said cosine similarity alone gives weak relevance at scale and recommended hybrid search plus reranking.

How do you measure RAG quality at scale?

Ramchandani recommended building evaluation sets from real user questions, testing ranking changes against a static cluster, and sampling live traffic.

How can teams control embedding costs?

Ramchandani suggested training a smaller student model and exploring binary or Matryoshka-style embeddings; Bratseth said these make ColBERT economic at scale.

This article is based on remarks by Jon Bratseth, CEO and Founder, Vespa.ai; Aakarsh Ramchandani, Chief Strategy Officer, RavenPack at AI Connect webinar (2024) on October 30, 2024. More coverage: Webinars.

Filed under: Data & AI Infrastructure, AI Use Cases & Applications, Observability & Data Quality