A PM asks why the RAG assistant feels sluggish next to a plain chatbot: account for the extra latency and what you'd do

Short answer·Medium·4.0 · 0·~3 min·Asked atCohere Humanloop Netflix

Attempt it

A product manager notices that your RAG assistant feels noticeably slower to respond than a plain LLM chatbot, and asks you to explain where the extra time goes and how to make it feel faster. Break down the additional latency RAG introduces and list the mitigations you would apply without removing retrieval.

Free · 2 AI evals / day

Start by comparing the two paths. A plain chatbot does one thing: take the user's message and generate. A RAG assistant does several things *before* it can generate. Walk the extra stages. First it **embeds the query** into a vector. Then it runs a search — vector search, often plus keyword search — over the index. Then it optionally reranks the candidates with a cross-encoder. Then it assembles a prompt that now contains the retrieved chunks. That last part matters: a longer prompt means more input tokens the model has to read (the prefill step) before it can produce the first output token. So the user waits for retrieval *and* a heavier prefill, on top of the same generation a plain chatbot does. The fix is to hide and shrink that overhead — not to remove retrieval, which keeps answers grounded. The biggest perceived win is **streaming**: show tokens the moment generation starts. Then make the work smaller: cache query embeddings and results, run retrieval steps in parallel, trim how many chunks you keep, and skip the reranker on easy queries. The message for the PM: the extra time buys accuracy, and most of it can be made invisible.

A plain chatbot goes message → generate. RAG inserts a pipeline in front: **embed the query**, then run vector (and often keyword) search. It optionally reranks the candidate set with a cross-encoder, then assembles a prompt that now carries the retrieved chunks. Each stage is real wall-clock time the plain chatbot never spends. The part teams forget is the prompt itself. Those retrieved chunks make the input far longer, and the model must process every input token in the **prefill** phase before it emits the first output token. So RAG's latency is retrieval time plus a heavier prefill plus the same generation — and the prefill cost scales with how much context you stuffed in. The fix is to hide and shrink the overhead, never to drop retrieval. The single biggest perceived-latency win is **streaming**: the user sees tokens the instant generation starts, so the wait feels like a fast chatbot even though work happened first. Then shrink the real cost. Run retrieval steps in parallel where they don't depend on each other. Cache aggressively — query embeddings, retrieval results, and full answers for repeated or paraphrased queries via semantic caching. Trim top-k and compress retrieved chunks so the prompt is smaller and prefill is cheaper. Skip the reranker on easy queries and consider a faster ANN configuration. The framing for the PM: the extra latency buys grounding and freshness, and most of it can be made invisible with streaming plus caching.

TL;DR

RAG adds embedding, search, optional reranking, and a longer prompt (heavier prefill) before generation. Stream to hide it, then cache, parallelize, and trim top-k to shrink it — don't drop retrieval.

Memory aid

Easy to grasp

Imagine two waiters. One takes your order and walks straight to the kitchen. The other first runs to the library to look up the recipe, picks the best version, copies it out, and only then heads to the kitchen. The second waiter is your RAG assistant — those extra errands are real time. You can't skip the library trip (it's what keeps the food accurate), but you can have the waiter start bringing out dishes the moment they're ready instead of waiting for the whole meal, run the errands at the same time, and keep a copy of recipes they've looked up before. RAG, short for Retrieval-Augmented Generation, gets faster the same way.

Key concepts

Retrieval-Augmented Generation Inference Optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

"Why does our RAG assistant feel slower than ChatGPT?" is a question every PM eventually asks, and the wrong answer — "let's drop retrieval" — throws away the entire reason the system exists. The right answer starts by being precise about where the time actually goes, because RAG latency is not one number; it's a chain of stages a plain chatbot simply doesn't run.

The interview value here is twofold: can you decompose the pipeline into the stages that add wall-clock time, including the easy to miss prefill cost of a longer prompt, and can you separate making it feel faster from making it actually faster? This deep dive walks the latency anatomy, the perceived versus actual mitigation split, the tradeoffs each fix carries, and how to measure it so the PM sees real progress instead of a moved mean.

Where the extra time actually goes

A plain chatbot has a two-step critical path: receive the message, generate a reply. RAG inserts a pipeline in front of generation, and every stage is wall-clock the chatbot never spends.

Embed the query. The user's text is encoded into a vector — one model call, typically tens of milliseconds, or near-zero if the embedding is cached.

Search. The query vector is matched against the index via approximate nearest neighbor search, often alongside a BM25 keyword search. Usually fast, but the tail depends on index type, corpus size, and the recall target you've configured.

Rerank (optional). A cross-encoder reads the query with each candidate and rescores. This is a forward pass per candidate, so its cost scales with the candidate-pool size N.

Assemble the prompt. The retrieved chunks are concatenated into the context. This produces a much longer input than the bare question — and that sets up the cost the next section covers.

The prefill tax everyone forgets

Perceived vs actual: streaming first, then shrink the work

Measuring it and the tradeoffs that bite

Caching: the layer that pays back the most

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Perplexity streams its answer while showing a retrieval/search indicator so the wait reads as progress, not lag
Pinecone and Qdrant expose ANN parameters (ef_search, nprobe) to trade a little recall for lower tail latency

What an interviewer would ask next. Try answering before peeking at the approach.

QIf streaming hides the wait, why bother shrinking the actual latency at all?

Streaming only hides the time before the first token; it doesn't cut cost, server load, or the tail. A high time to first token still feels laggy even when streaming, and total wall-clock drives throughput and bill. So you stream for perception and trim prefill, parallelize, and cache for cost and the p99 the streaming illusion can't paper over.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reaching for 'just drop retrieval to make it fast' — which throws away the grounding and freshness RAG exists to provide, when most of the latency can be hidden with streaming instead.

60 second bullets to scan on the way to the call.

List the stages RAG adds before generation that a plain chatbot skips
Explain why the retrieved context makes prefill heavier and delays the first token

Primary sources. Browse if you want the original framing.