Design a production RAG system serving 10,000 concurrent users over a 100M-document corpus, with a P95 latency target of 2 seconds. Walk through the architecture and the bottleneck at each layer.
Design a production RAG system: 100M documents in the corpus, 10,000 concurrent users, P95 end to end latency of 2 seconds, freshness requirement of 1 hour for new documents. Walk through the architecture, the bottleneck at each layer, and the key capacity decisions. Use 2026-current systems (modern embedding models, modern LLMs, modern vector DBs).
Free · 2 AI evals / day
A production RAG system at this scale has five layers. The ingestion pipeline brings new documents in, chunks them, embeds them, and writes them to a vector database. The index stores those embeddings and serves nearest-neighbor lookups. The retrieval pipeline runs at query time: embed the question, search hybrid (BM25 plus dense), then rerank.
The generation layer passes the top chunks plus the query to a frontier LLM like Claude Opus 4.7 or GPT-5.5. The serving layer wraps everything behind a load balancer that handles 10,000 concurrent users.
The critical insight is the 2-second budget. The LLM call eats most of it: roughly 1.2 to 1.5 seconds. That leaves only 500 milliseconds for embedding, retrieval, and reranking combined. Streaming the LLM output helps users feel responsiveness faster than the full P95 number suggests.
At 100M docs you also have storage and freshness problems: quantization shrinks the index, and a streaming ingestion queue keeps it within one hour of the latest documents.
**RAG at this scale** is five layers, each with its own latency, cost, and freshness story.
The ingestion layer pulls new docs off a queue (Kafka, SQS) and chunks them at 256-512 tokens with overlap. It embeds with a high-throughput model like Voyage Embed v3 or Cohere Embed v4 via batch endpoints, then upserts into a managed vector DB. The 1-hour freshness SLA rules out nightly batch.
The index holds roughly 300M vectors (100M docs times ~3 chunks each) on sharded HNSW in Pinecone, Qdrant Cloud, or Milvus. At 3072 dims that is ~3.4 TB raw, so product quantization or scalar quantization is required, trading 1-3 points of recall for 4-8x storage savings.
The per-query budget: 50 ms to embed, 50-100 ms hybrid BM25+dense with RRF, 100-150 ms for a cross-encoder rerank like Cohere Rerank or BGE-reranker over the top 50. That is ~300 ms of retrieval, leaving ~1.3 s for the generator (Claude Opus 4.7, GPT-5.5, or self-hosted Llama 4 on H200/B200).
**The LLM dominates** both latency and cost. Mitigations: prompt caching, KV cache reuse, and small model routing for easy queries. Capacity at 10K concurrent users is bounded by LLM throughput, so plan on provisioned throughput or self-hosted clusters.
Quick reference
TL;DR
Five layers: ingest, index, retrieve, generate, serve. The LLM call eats ~1.3s of your 2s budget. Scale-out levers are quantization, hybrid retrieval, prompt caching, and generator routing.
Imagine a giant library with 100 million pages and 10,000 readers asking questions at the same time. You cannot let any reader wait more than 2 seconds. You build a conveyor belt with five stations. Station 1 keeps the library up to date as new books arrive. Station 2 stores tiny number summaries of every page so search is fast. Station 3 takes a reader's question, finds the most likely pages, and re-ranks the top results. Station 4 hands those pages and the question to a smart assistant that writes the answer. Station 5 is the front desk that talks to the readers. The slowest station is the smart assistant: it eats most of the 2 seconds. The rest of the work is making sure the assistant gets only the best pages and that nobody has to wait in line for it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Corpus is small (under 100K docs) and traffic is light A monolithic single-process retriever + LLM call is cheaper to operate; the five-layer split adds operational overhead you do not need.
Latency SLA is sub-500 ms end to end The LLM call alone exceeds 500 ms P95 for any frontier model; you need a different architecture, likely a small fine-tuned model or extractive QA rather than generative RAG.
Answers must be deterministic and auditable for compliance Generative RAG can paraphrase or hallucinate; a structured KB lookup with templated answers may be a better fit for highly regulated domains.
Documents change every few seconds Streaming embed + index upsert at high write rate can saturate the vector DB; consider a hybrid where recent docs are served from a hot fulltext index and older docs from the vector store.
Quick scan · 2–4 min
Everything important, quickly.
10 min: walk all five layers, do the index sizing math out loud, name the latency budget per stage, and rank the three or four highest leverage cost levers.
Code example
python
# Sketch of a 2026 production RAG query handler.
from anthropic import Anthropic
from pinecone import Pinecone
from cohere import Client as CohereClient
client = Anthropic()
pc = Pinecone().Index("corpus-v4")
co = CohereClient()
async def answer(query: str) -> str:
# 1. Embed the query (~50 ms).
q_emb = (await co.embed_async(texts=[query], model="embed-v4")).embeddings[0]
# 2. Hybrid retrieve: dense + BM25 fused with RRF (~80 ms).
dense = pc.query(vector=q_emb, top_k=50, include_metadata=True).matches
bm25 = pc.query_sparse(query=query, top_k=50).matches
candidates = rrf_fuse([dense, bm25], k=60)[:50]
# 3. Cross-encoder rerank to top 8 (~120 ms).
reranked = co.rerank(query=query, documents=candidates, top_n=8, model="rerank-v4")
# 4. Generate with prompt caching on system prompt and shared chunks (~1.3 s P95).
return await client.messages.create(
model="claude-opus-4-7",
system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user", "content": render(query, reranked)}],
stream=True,
)
Side-by-side
Layer
Latency budget
Scale-out lever
Dominant cost
Ingestion
async, <1 h SLA
queue + batch embed endpoints
embedding API
Vector index
~50 ms retrieval
shard + product quantization
storage + read replicas
Retrieval pipeline
~300 ms total
hybrid + reranker top-50→top-8
reranker calls
Generation
~1.3 s P95
prompt caching + routing
LLM tokens
Serving
stateless
provisioned throughput or self-host
concurrent LLM capacity
Real-world examples
Real products, models, and research that use this idea.
1Notion AI runs hybrid retrieval over a per-workspace index in Pinecone serverless tiering, with generator routing between Claude Haiku 4.7 and Claude Opus 4.7.
2Perplexity uses streaming ingest into a sharded vector index, hybrid retrieval with RRF, and a frontier model generator with aggressive prompt caching.
3GitHub Copilot Workspace embeds repository chunks via a high-throughput endpoint, indexes per-repo, and routes between a small and large generator based on query complexity.
4Anthropic Claude 4.7 contextual retrieval ships prompt caching as a first class feature, making per-chunk context cheap to repeat at query time.
5Pinecone serverless documents the exact storage tiering and quantization story used by customers running 100M+ vector corpora at P95 sub-second retrieval.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere do you put a global semantic answer cache, and how do you prevent it from serving stale answers when the corpus updates?
Sign in or go Pro to see the approach for this follow-up.
QHow does the architecture change if the corpus grows to 1 billion documents?
Sign in or go Pro to see the approach for this follow-up.
QWalk through how you would shrink the generator latency tail without changing the model.
Sign in or go Pro to see the approach for this follow-up.
QWhat is your eval gate before any of these scale out changes ships?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating RAG as one big black box. Latency, cost, and freshness are dominated by different layers; you have to budget each one separately or you cannot hit a P95 SLA.