Retrieval P95 latency tripled after the corpus grew from 1M to 20M vectors: diagnose it
After your corpus grew from 1M to 20M vectors, retrieval P95 latency roughly tripled even though query volume is unchanged. Diagnose the likely causes and describe how you would bring latency back down without tanking recall.
Latency tracking corpus size points at the ANN index: params tuned for 1M over-explore at 20M, and the index may have spilled out of RAM. Fix: re-tune to a recall target, shard, quantize — not just bigger hardware.
Imagine a library that grew from one room to twenty rooms overnight, but the same number of visitors come each day. Each visitor now walks much farther to find a book, so the average wait time triples even though nobody new showed up. Two things went wrong. First, the search instructions you wrote for one room — "check these few shelves" — now make people wander far more shelves to be just as sure they found the right book. Second, the catalog that used to sit on the front desk no longer fits, so it got moved to a back basement, and every lookup means a slow trip downstairs. The fixes are to rewrite the search instructions for the bigger building, split the books across several buildings searched at the same time, and shrink the catalog so it fits back on the desk.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Production debugging questions reward a method, not a memorized cause. The single most useful observation here is given to you in the prompt: query volume is flat, the corpus grew 20x, and latency tracked the corpus, not the traffic. That correlation is a scalpel. It cuts away every component whose cost depends on request rate — the web tier, connection pools, the LLM call — and leaves the one thing whose cost depends on corpus size: the approximate nearest neighbor search inside the vector store.
From there the senior move is to resist the urge to act before measuring. The two real causes — stale index parameters and the index falling out of memory — look similar from the outside but demand different fixes, and the wrong fix wastes money. This deep dive walks the diagnostic path, explains why each cause produces the symptom, and orders the remedies so you fix the algorithmic cost before you size the hardware.
Reading the symptom: why the index is the suspect
Start with what changed and what did not. Query volume is unchanged, so this is not a load or concurrency problem — connection limits, thread pools, and autoscaling are off the table. What changed is corpus size, 20x. Latency moved with it, roughly 3x at P95.
That pairing is diagnostic. Most of a RAG request's cost is independent of corpus size. Embedding the query is fixed work. The LLM generation depends on prompt and output length, not on how many documents exist. Network hops are constant. The one stage whose cost grows with the number of stored vectors is the nearest neighbor search.
The first concrete action is to confirm that by isolating the retrieval call. Wrap a timer around just the vector search, separate from query embedding and the LLM call, and look at P95, not the mean. The tail is where the interesting behavior hides: a mean that drifts up suggests more work per query, while a tail that jumps in steps suggests something discontinuous like paging. Splitting the latency budget into its stages turns a vague 'it got slow' into a specific 'the search call went from 12 ms to 38 ms at P95,' which is what every later decision keys off.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- HNSW indexes in FAISS, Qdrant, and Weaviate degrade sharply once the graph no longer fits in RAM and queries start paging from disk.
- IVF-PQ in FAISS is the canonical fix for footprint: product quantization shrinks vectors so a large index fits in memory, with a rerank pass recovering recall.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set ef_search and nprobe to a target recall instead of guessing?
Sweep the parameter on a labeled query set, plot recall@k against latency, and pick the smallest value that meets the recall target — re-running the sweep whenever the corpus size changes materially.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Throwing a bigger box at it without re-tuning — you pay for capacity while the search still over-explores far more candidates than it needs.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.