In a production RAG system with a 2-second end to end latency budget for non-streaming responses, which step typically dominates and where should the optimization effort go?
A RAG request runs five steps in sequence: embed the query, search the vector database, optionally rerank the candidates, build the prompt, then let the **LLM** generate the answer. To find the bottleneck, add up how long each step takes.
The first four are fast. Embedding the query is one API call, roughly 50-100ms. Vector search with a good index is sub-100ms even over millions of documents, because that is exactly what approximate nearest neighbor search is built for. Reranking the top candidates adds maybe 100-200ms. Assembling the prompt is just string building, under 10ms.
The last step is the slow one. The model produces the answer one token at a time, and a few hundred tokens take well over a second. So out of a 2-second budget, generation alone is 1.2-1.8 seconds and everything else shares 200-500ms.
That tells you where to spend effort. Tuning the vector database buys you milliseconds. Shrinking the prompt, picking a faster model, or streaming the answer so the user sees words immediately is where the real time lives.
**LLM generation** dominates a 2-second RAG budget. The realistic breakdown with managed services is: query embedding 50-100ms, vector retrieval 50-100ms, reranking 100-200ms, prompt assembly under 10ms, and generation 1.2-1.8 seconds. Generation outweighs every other step by an order of magnitude.
The reason is structural. Generation is autoregressive: each output token depends on all previous ones, so a 300-token answer is 300 sequential forward passes. Retrieval is a single approximate nearest neighbor lookup that an HNSW index serves in tens of milliseconds regardless of corpus size, which is the entire point of ANN.
So the optimization effort goes to generation. The biggest lever is streaming: it drops time to first token to a few hundred ms and transforms perceived latency even when total time is unchanged. Next, cut output and prompt tokens with fewer chunks and a tighter system prompt. Then pick a smaller or faster model for the easy queries, and exploit prefix caching on the static prompt prefix.
The trap is optimizing the cheap steps. Swapping vector databases or tuning the reranker shaves milliseconds off a budget where seconds live in generation.
Quick reference
TL;DR
In a 2s RAG budget, LLM generation eats 1.2-1.8s; embed, retrieve, rerank, and augment together fit in 200-500ms. Optimize generation, not the retrieval stack.
Imagine a 2-second relay race with five runners. Four of them sprint their legs in a fraction of a second each: looking up the question's meaning, searching the library, reordering the best books, and gluing the pages into a packet. Then the fifth runner has to read the packet out loud, word by word, before anyone gets the answer. That last runner takes more than a full second on his own. If you want the race to finish faster, there's no point yelling at the four fast runners. You speed up the one who reads out loud: hand him a shorter packet, swap him for a quicker reader, or let him start speaking the moment he has the first word. In a RAG system the slow reader is the language model generating tokens one at a time.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Streaming-first chat UI where perceived latency is what matters Total wall clock is less relevant; time to first token via streaming is the metric to optimize, not the 2s end to end sum.
Batch or offline RAG pipelines processing documents asynchronously Per-request latency is irrelevant; throughput and cost per document dominate, so the per-step budget framing does not apply.
Tiny self-hosted models on constrained hardware with slow retrieval On weak setups an unindexed brute force search can rival generation, so the generation dominates assumption must be re-measured.
Quick scan · 2–4 min
Everything important, quickly.
3 min: enumerate the five steps with latencies, explain why generation is autoregressive and dominant, then list the generation side optimization levers in priority order.
Side-by-side
RAG step
Typical latency
Scales with
Optimization headroom
Query embedding
50-100ms
Fixed (one forward pass)
Low
Vector retrieval (ANN)
50-100ms
Barely (HNSW is log ish)
Low
Reranking (cross-encoder)
100-200ms
Top-k candidate count
Low-medium
Prompt assembly
<10ms
Negligible
None
LLM generation
1.2-1.8s
Output tokens + prompt size
High
Real-world examples
Real products, models, and research that use this idea.
1Perplexity streams its answer token by token so users read the response forming while generation is still in flight, hiding the 1s+ generation cost.
2OpenAI's Assistants and Responses APIs default to server sent event streaming precisely because non-streaming RAG answers feel slow on generation.
3Anthropic's Claude API exposes prompt caching so the static RAG prompt prefix skips re-prefill, cutting time to first token on repeated queries.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you must keep total wall clock under 2s and the answer is genuinely long, what do you change first?
Sign in or go Pro to see the approach for this follow-up.
QHow does the latency picture change when you move from managed APIs to self-hosted generation?
Sign in or go Pro to see the approach for this follow-up.
QWhere does the breakdown shift if retrieval involves multi-hop or agentic loops?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the vector database is the bottleneck. Well-tuned ANN retrieval is sub-100ms even at 10M docs; token by token generation is the part measured in seconds.