After your corpus grew from 1M to 20M vectors, retrieval P95 latency roughly tripled even though query volume is unchanged. Diagnose the likely causes and describe how you would bring latency back down without tanking recall.
Free · 2 AI evals / day
The clue is that latency tracked corpus size while query volume stayed flat. That points straight at the component whose cost depends on how much data you have: the ANN index in your vector store. The LLM and the network do not care how big the corpus is.
Two causes dominate. First, the index parameters were tuned for 1 million vectors. At 20 million, the same recall target forces the search to explore far more candidates per query, so each search does more work and takes longer. Second, the index may have outgrown memory. What fit in RAM at 1M may spill to disk at 20M, and every query then pages from storage, which is much slower.
To confirm, time the retrieval call by itself to rule out the LLM and network, check whether the index is fully in RAM, and measure recall versus latency at the new size.
The fixes, in order: re-tune the index parameters for 20M against a recall target, shard the index across nodes so each searches a smaller slice in parallel, and quantize the vectors so the index fits back in RAM. Do not just buy a bigger machine without re-tuning.
Latency scaling with corpus size while query volume is flat localizes the problem to the **ANN index**, not the application, the LLM, or the network. Two failure modes dominate at 20x growth.
First, the index parameters were tuned for 1M and no longer hold. HNSW `ef_search` or IVF `nprobe` were set for a smaller graph or cluster count. At 20M the search has to traverse a larger structure and explore far more candidates per query to hit the same recall, so per-query work climbs. Second, you may have outgrown memory. An index that fit in RAM at 1M can spill to disk at 20M. Each query then pages from storage, which makes latency jump in steps rather than rising smoothly.
To diagnose, isolate retrieval by timing the search call alone and confirm the index is fully memory-resident. Then plot the recall@k versus latency curve at the new size to see how hard the search is working.
The fixes come roughly in order. Re-tune the parameters for 20M against an explicit recall target so you stop over-exploring. Shard the index across nodes so each shard searches a smaller slice in parallel and you fan out. Then cut the memory footprint with vector quantization, accepting a small recall hit you recover with a rerank pass over the top candidates. The anti-pattern is bumping the hardware blindly — you pay for capacity while the search still over-explores.
Quick reference
TL;DR
Latency tracking corpus size points at the ANN index: params tuned for 1M over-explore at 20M, and the index may have spilled out of RAM. Fix: re-tune to a recall target, shard, quantize — not just bigger hardware.
Imagine a library that grew from one room to twenty rooms overnight, but the same number of visitors come each day. Each visitor now walks much farther to find a book, so the average wait time triples even though nobody new showed up. Two things went wrong. First, the search instructions you wrote for one room — "check these few shelves" — now make people wander far more shelves to be just as sure they found the right book. Second, the catalog that used to sit on the front desk no longer fits, so it got moved to a back basement, and every lookup means a slow trip downstairs. The fixes are to rewrite the search instructions for the bigger building, split the books across several buildings searched at the same time, and shrink the catalog so it fits back on the desk.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Quantizing aggressively on a corpus where recall is already marginal Product quantization adds approximation error; on a borderline-recall corpus the rerank pass may not recover enough, hurting answer quality.
Sharding a corpus small enough to fit one well-tuned node Sharding adds merge overhead and operational complexity that is not worth it until per-node memory or search cost is the real bottleneck.
Re-tuning ef_search alone when the index has clearly spilled to disk Parameter tuning cannot beat disk paging; the memory residency problem must be fixed first via quantization or more RAM.
Quick scan · 2–4 min
Everything important, quickly.
4 min: localize to the ANN index, walk the stale-params and memory-spill causes, describe the isolation diagnosis, then prioritize re-tune, shard, quantize over buying hardware.
Real-world examples
Real products, models, and research that use this idea.
1HNSW indexes in FAISS, Qdrant, and Weaviate degrade sharply once the graph no longer fits in RAM and queries start paging from disk.
2IVF-PQ in FAISS is the canonical fix for footprint: product quantization shrinks vectors so a large index fits in memory, with a rerank pass recovering recall.
3Managed stores like Pinecone shard the index across pods so each pod searches a smaller slice and queries fan out in parallel.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set ef_search and nprobe to a target recall instead of guessing?
Sign in or go Pro to see the approach for this follow-up.
QWhy does an HNSW index degrade so sharply once it no longer fits in RAM?
Sign in or go Pro to see the approach for this follow-up.
QHow much recall does product quantization cost, and how do you buy it back?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Throwing a bigger box at it without re-tuning — you pay for capacity while the search still over-explores far more candidates than it needs.