In a production RAG system with a 2 second end to end latency budget for non-streaming responses, which step typically dominates and where should the optimization effort go?
In a 2s RAG budget, LLM generation eats 1.2-1.8s; embed, retrieve, rerank, and augment together fit in 200-500ms. Optimize generation, not the retrieval stack.
Imagine a 2-second relay race with five runners. Four of them sprint their legs in a fraction of a second each: looking up the question's meaning, searching the library, reordering the best books, and gluing the pages into a packet. Then the fifth runner has to read the packet out loud, word by word, before anyone gets the answer. That last runner takes more than a full second on his own. If you want the race to finish faster, there's no point yelling at the four fast runners. You speed up the one who reads out loud: hand him a shorter packet, swap him for a quicker reader, or let him start speaking the moment he has the first word. In a RAG system the slow reader is the language model generating tokens one at a time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Latency is where RAG interviews separate people who have shipped a system from people who have read a tutorial. The slogan answer is that you embed, retrieve, rerank, augment, and generate. The interview question is sharper: given a fixed 2-second budget for a non-streaming response, which of those five steps actually consumes the time, and where do you spend your engineering effort?
The honest answer is that the budget is wildly lopsided. Four of the five steps are fast and roughly constant time. One step, generation, is autoregressive and runs in seconds. A candidate who reaches for the vector database as the bottleneck has usually never profiled a real request; a candidate who names generation and can say why has almost certainly shipped one.
This deep dive builds the budget from first principles, explains why each step costs what it costs, walks the optimization levers in priority order, and then shows when the standard breakdown stops holding. Knowing where the time goes is only useful if you also know what to do about it, so every section ends pointing back at the lever it unlocks.
Building the budget from first principles
Start by writing down what each step physically does, because the cost follows directly from the mechanism and the millisecond figures stop being something to memorize.
Query embedding is a single forward pass through an embedding model that turns the query string into one vector. One round trip to a managed embedding API lands in 50-100ms, most of which is network rather than compute. It does not grow with your corpus, only with prompt length, and queries are short, so this term is effectively a constant.
Vector retrieval is one approximate nearest neighbor lookup. An HNSW graph walks a small number of hops through a layered proximity graph to find the top candidates, so lookup time grows roughly logarithmically with corpus size, not linearly. That is the entire reason ANN exists: you trade a sliver of recall for a search that stays fast as the index grows. Even at ten million documents, a tuned index answers in well under 100ms, and going to a hundred million barely moves it.
Reranking runs a cross-encoder over the top-k candidates from retrieval. Because it scores query-document pairs jointly in a single model rather than comparing two precomputed vectors, it is heavier than the bi-encoder embedding, but it only ever sees a handful of candidates, so it lands at 100-200ms for a top-20 rerank. Prompt assembly is pure string concatenation, under 10ms, and never the problem. The headline: the first four steps are bounded, predictable, and cheap, and none of them scales in a way that threatens a 2-second budget.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| RAG step | Typical latency | Scales with | Optimization headroom |
|---|---|---|---|
| Query embedding | 50-100ms | Fixed (one forward pass) | Low |
| Vector retrieval (ANN) | 50-100ms | Barely (HNSW is log ish) | Low |
| Reranking (cross-encoder) | 100-200ms | Top-k candidate count | Low-medium |
| Prompt assembly | <10ms | Negligible | None |
| LLM generation | 1.2-1.8s | Output tokens + prompt size | High |
Real products, models, and research that use this idea.
- Perplexity streams its answer token by token so users read the response forming while generation is still in flight, hiding the 1s+ generation cost.
- OpenAI's Assistants and Responses APIs default to server sent event streaming precisely because non-streaming RAG answers feel slow on generation.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you must keep total wall clock under 2s and the answer is genuinely long, what do you change first?
Attack token count: cap max-tokens, trim retrieved context, switch to a faster decode model, and consider speculative decoding; streaming alone won't fix a hard total time SLA.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the vector database is the bottleneck. Well-tuned ANN retrieval is sub-100ms even at 10M docs; token by token generation is the part measured in seconds.
60 second bullets to scan on the way to the call.
The five RAG steps in order and the rough latency of each
Why autoregressive generation is the only step measured in seconds
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.