Design a production RAG system serving 10,000 concurrent users over a 100M document corpus, with a P95 latency target of 2 seconds. Walk through the architecture and the bottleneck at each layer.
Design a production RAG system: 100M documents in the corpus, 10,000 concurrent users, P95 end to end latency of 2 seconds, freshness requirement of 1 hour for new documents. Walk through the architecture, the bottleneck at each layer, and the key capacity decisions. Use 2026 current systems (modern embedding models, modern LLMs, modern vector DBs).
Five layers: ingest, index, retrieve, generate, serve. The LLM call eats ~1.3s of your 2s budget. Scale-out levers are quantization, hybrid retrieval, prompt caching, and generator routing.
Imagine a giant library with 100 million pages and 10,000 readers asking questions at the same time. You cannot let any reader wait more than 2 seconds. You build a conveyor belt with five stations. Station 1 keeps the library up to date as new books arrive. Station 2 stores tiny number summaries of every page so search is fast. Station 3 takes a reader's question, finds the most likely pages, and re-ranks the top results. Station 4 hands those pages and the question to a smart assistant that writes the answer. Station 5 is the front desk that talks to the readers. The slowest station is the smart assistant: it eats most of the 2 seconds. The rest of the work is making sure the assistant gets only the best pages and that nobody has to wait in line for it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A production RAG system at 100M documents and 10,000 concurrent users is not a single design problem. It is five design problems stacked on top of each other, each with its own latency budget, freshness story, and cost driver. The senior interviewer is checking whether you can keep all five conversations alive in your head at once and rank them by where the leverage actually is.
The trap candidates fall into is to list components in a flat sequence: 'we have a vector DB, we have embeddings, we have an LLM.' That gets a polite nod and a low score. The signal a staff level interviewer is hunting for is whether you can budget the 2-second P95 across stages, do the sizing math for 100M docs out loud, and explain which optimizations are zero risk versus which trade quality.
This deep dive walks the architecture end to end, calls out the bottleneck and the mitigation at each layer, and closes with how you would defend the design in a CFO room when the bill arrives.
The five-layer architecture
Production RAG decomposes cleanly into ingestion, indexing, retrieval, generation, and serving. Each layer has a single concern, and each layer has a different bottleneck.
Ingestion brings new documents into the system. The 1-hour freshness SLA means a queue-driven design: Kafka or SQS in front of a worker pool that chunks (semantic chunking or fixed-size 256-512 tokens with overlap), embeds via a high-throughput model like Voyage Embed v3 or Cohere Embed v4, and upserts to the vector DB. Batch endpoints amortize cost, and queue depth absorbs ingest bursts. Nightly batch is structurally disqualified by the 1-hour SLA.
Indexing holds the embeddings and answers nearest-neighbor lookups. At 100M docs and roughly 3 chunks per doc, that is ~300M vectors. Sharded HNSW in a managed system (Pinecone, Qdrant Cloud, Milvus, Vespa) is the standard answer in 2026, sharded by tenant or by hash. Latency target: ~50 ms P95 on top-50 retrieval.
Retrieval is the per-query pipeline: query embedding, hybrid BM25+dense with RRF fusion, then a cross-encoder rerank over the top 50 down to the top 8 or so chunks. Generation passes those chunks plus the query to a frontier LLM. Serving is the stateless API layer in front of all of it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Sketch of a 2026 production RAG query handler.
from anthropic import Anthropic
from pinecone import Pinecone
from cohere import Client as CohereClient
client = Anthropic()
pc = Pinecone().Index("corpus-v4")
co = CohereClient()
async def answer(query: str) -> str:
# 1. Embed the query (~50 ms).
q_emb = (await co.embed_async(texts=[query], model="embed-v4")).embeddings[0]
# 2. Hybrid retrieve: dense + BM25 fused with RRF (~80 ms).
dense = pc.query(vector=q_emb, top_k=50, include_metadata=True).matches
bm25 = pc.query_sparse(query=query, top_k=50).matches
candidates = rrf_fuse([dense, bm25], k=60)[:50]
# 3. Cross-encoder rerank to top 8 (~120 ms).
reranked = co.rerank(query=query, documents=candidates, top_n=8, model="rerank-v4")
# 4. Generate with prompt caching on system prompt and shared chunks (~1.3 s P95).
return await client.messages.create(
model="claude-opus-4-7",
system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user", "content": render(query, reranked)}],
stream=True,
)| Layer | Latency budget | Scale-out lever | Dominant cost |
|---|---|---|---|
| Ingestion | async, <1 h SLA | queue + batch embed endpoints | embedding API |
| Vector index | ~50 ms retrieval | shard + product quantization | storage + read replicas |
| Retrieval pipeline | ~300 ms total | hybrid + reranker top-50→top-8 | reranker calls |
| Generation | ~1.3 s P95 | prompt caching + routing | LLM tokens |
| Serving | stateless | provisioned throughput or self-host | concurrent LLM capacity |
Real products, models, and research that use this idea.
- Notion AI runs hybrid retrieval over a per-workspace index in Pinecone serverless tiering, with generator routing between Claude Haiku 4.7 and Claude Opus 4.7.
- Perplexity uses streaming ingest into a sharded vector index, hybrid retrieval with RRF, and a frontier model generator with aggressive prompt caching.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere do you put a global semantic answer cache, and how do you prevent it from serving stale answers when the corpus updates?
Cache keyed by query embedding similarity; invalidate by corpus epoch and TTL. Track cache hit rate per query class. Stale-on-update is the failure mode you guard against with epoch tags on cached entries.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating RAG as one big black box. Latency, cost, and freshness are dominated by different layers; you have to budget each one separately or you cannot hit a P95 SLA.
60 second bullets to scan on the way to the call.
The five-layer decomposition and what each layer owns
Latency budget per stage and why the LLM dominates
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.