Design a production RAG system serving 10,000 concurrent users over a 100M-document corpus, with a P95 latency target of 2 seconds. Walk through the architecture and the bottleneck at each layer.
Design a production RAG system: 100M documents in the corpus, 10,000 concurrent users, P95 end to end latency of 2 seconds, freshness requirement of 1 hour for new documents. Walk through the architecture, the bottleneck at each layer, and the key capacity decisions. Use 2026-current systems (modern embedding models, modern LLMs, modern vector DBs).
Five layers: ingest, index, retrieve, generate, serve. The LLM call eats ~1.3s of your 2s budget. Scale-out levers are quantization, hybrid retrieval, prompt caching, and generator routing.
Imagine a giant library with 100 million pages and 10,000 readers asking questions at the same time. You cannot let any reader wait more than 2 seconds. You build a conveyor belt with five stations. Station 1 keeps the library up to date as new books arrive. Station 2 stores tiny number summaries of every page so search is fast. Station 3 takes a reader's question, finds the most likely pages, and re-ranks the top results. Station 4 hands those pages and the question to a smart assistant that writes the answer. Station 5 is the front desk that talks to the readers. The slowest station is the smart assistant: it eats most of the 2 seconds. The rest of the work is making sure the assistant gets only the best pages and that nobody has to wait in line for it.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
10 min: walk all five layers, do the index sizing math out loud, name the latency budget per stage, and rank the three or four highest leverage cost levers.
# Sketch of a 2026 production RAG query handler.
from anthropic import Anthropic
from pinecone import Pinecone
from cohere import Client as CohereClient
client = Anthropic()
pc = Pinecone().Index("corpus-v4")
co = CohereClient()
async def answer(query: str) -> str:
# 1. Embed the query (~50 ms).
q_emb = (await co.embed_async(texts=[query], model="embed-v4")).embeddings[0]
# 2. Hybrid retrieve: dense + BM25 fused with RRF (~80 ms).
dense = pc.query(vector=q_emb, top_k=50, include_metadata=True).matches
bm25 = pc.query_sparse(query=query, top_k=50).matches
candidates = rrf_fuse([dense, bm25], k=60)[:50]
# 3. Cross-encoder rerank to top 8 (~120 ms).
reranked = co.rerank(query=query, documents=candidates, top_n=8, model="rerank-v4")
# 4. Generate with prompt caching on system prompt and shared chunks (~1.3 s P95).
return await client.messages.create(
model="claude-opus-4-7",
system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user", "content": render(query, reranked)}],
stream=True,
)| Layer | Latency budget | Scale-out lever | Dominant cost |
|---|---|---|---|
| Ingestion | async, <1 h SLA | queue + batch embed endpoints | embedding API |
| Vector index | ~50 ms retrieval | shard + product quantization | storage + read replicas |
| Retrieval pipeline | ~300 ms total | hybrid + reranker top-50→top-8 | reranker calls |
| Generation | ~1.3 s P95 | prompt caching + routing | LLM tokens |
| Serving | stateless | provisioned throughput or self-host | concurrent LLM capacity |
Real products, models, and research that use this idea.
- Notion AI runs hybrid retrieval over a per-workspace index in Pinecone serverless tiering, with generator routing between Claude Haiku 4.7 and Claude Opus 4.7.
- Perplexity uses streaming ingest into a sharded vector index, hybrid retrieval with RRF, and a frontier model generator with aggressive prompt caching.
- GitHub Copilot Workspace embeds repository chunks via a high-throughput endpoint, indexes per-repo, and routes between a small and large generator based on query complexity.
- Anthropic Claude 4.7 contextual retrieval ships prompt caching as a first class feature, making per-chunk context cheap to repeat at query time.
- Pinecone serverless documents the exact storage tiering and quantization story used by customers running 100M+ vector corpora at P95 sub-second retrieval.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere do you put a global semantic answer cache, and how do you prevent it from serving stale answers when the corpus updates?
QHow does the architecture change if the corpus grows to 1 billion documents?
QWalk through how you would shrink the generator latency tail without changing the model.
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating RAG as one big black box. Latency, cost, and freshness are dominated by different layers; you have to budget each one separately or you cannot hit a P95 SLA.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.