In a production RAG system with a 2-second end to end latency budget for non-streaming responses, which step typically dominates and where should the optimization effort go?

A RAG request runs five steps in sequence: embed the query, search the vector database, optionally rerank the candidates, build the prompt, then let the **LLM** generate the answer. To find the bottleneck, add up how long each step takes. The first four are fast. Embedding the query is one API call, roughly 50-100ms. Vector search with a good index is sub-100ms even over millions of documents, because that is exactly what approximate nearest neighbor search is built for. Reranking the top candidates adds maybe 100-200ms. Assembling the prompt is just string building, under 10ms. The last step is the slow one. The model produces the answer one token at a time, and a few hundred tokens take well over a second. So out of a 2-second budget, generation alone is 1.2-1.8 seconds and everything else shares 200-500ms. That tells you where to spend effort. Tuning the vector database buys you milliseconds. Shrinking the prompt, picking a faster model, or streaming the answer so the user sees words immediately is where the real time lives.

**LLM generation** dominates a 2-second RAG budget. The realistic breakdown with managed services is: query embedding 50-100ms, vector retrieval 50-100ms, reranking 100-200ms, prompt assembly under 10ms, and generation 1.2-1.8 seconds. Generation outweighs every other step by an order of magnitude. The reason is structural. Generation is autoregressive: each output token depends on all previous ones, so a 300-token answer is 300 sequential forward passes. Retrieval is a single approximate nearest neighbor lookup that an HNSW index serves in tens of milliseconds regardless of corpus size, which is the entire point of ANN. So the optimization effort goes to generation. The biggest lever is streaming: it drops time to first token to a few hundred ms and transforms perceived latency even when total time is unchanged. Next, cut output and prompt tokens with fewer chunks and a tighter system prompt. Then pick a smaller or faster model for the easy queries, and exploit prefix caching on the static prompt prefix. The trap is optimizing the cheap steps. Swapping vector databases or tuning the reranker shaves milliseconds off a budget where seconds live in generation.