Your team is tuning top-k for the retriever in a production RAG system. What is the strongest reason NOT to just push top-k to its highest possible value?
Higher top-k costs more tokens and latency, then hurts faithfulness through lost-in-the-middle attention sag and precision dilution. Recall saturates around 5-15 while costs keep climbing.
Imagine asking a friend a question and handing them a stack of fifty printed pages they have to read before answering. Three of those pages contain the answer; the other forty seven are vaguely related. Your friend now has to find the three useful pages, ignore the rest, and keep their thinking sharp. Most people would do worse than if you had handed them just five pages that included the three good ones. They would take longer, get distracted, and sometimes pick a wrong fact from the noise. RAG works the same way. More retrieved chunks does not mean better answers past a point. Recall plateaus, latency climbs, and the model's attention spreads too thin across mostly irrelevant content.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Top-k looks like a single number, but tuning it well is a three-axis optimization problem. Push it too low and you miss relevant chunks. Push it too high and three different costs kick in: token spend, latency, and a quiet but real drop in faithfulness driven by lost-in-the-middle attention sag plus precision dilution from marginal chunks. The right shape for the curve is not 'more is always better' but a saturation followed by decline.
This deep dive walks through what happens to retrieval recall, attention behavior, and answer faithfulness as k grows; why the three curves overlap to create a relatively narrow sweet spot; and how the modern retrieve wide rerank narrow pattern decouples retrieval depth from prompt depth so you stop paying the dilution cost.
The recall curve saturates faster than people expect
Retrieval recall@k measures the fraction of ground-truth relevant chunks that appear in the retriever's top-k results. The curve almost always has the same shape: a steep climb at small k, then a knee, then a long plateau.
For most corpora with a competent embedding model, recall@10 captures 80-90 percent of what recall@100 will ever find. The chunks added between rank 10 and rank 100 are mostly from the same semantic neighborhood as the top hits, not new information. Doubling k from 10 to 20 typically lifts recall by 2-4 points; doubling from 20 to 40 lifts it by less than 1.
This saturation is the first reason a high k is wasteful. The marginal chunk you added at rank 47 is unlikely to be a genuinely new fact bearer. It is more likely a near-duplicate of something already at rank 5.
The shape changes when the corpus is unusually broad or the queries are unusually ambiguous. In those cases the long tail of k carries genuine new information, and a larger k is justified. The diagnostic is straightforward: plot recall@k on labeled data and locate the knee. If it sits at 5, your default should be 5. If it sits at 50, your retriever needs a reranker, not a larger prompt.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| top-k size | Recall | Latency / cost | Faithfulness | Use pattern |
|---|---|---|---|---|
| top-k = 3 | Often misses borderline cases | Cheapest | High when retrieval is good, brittle when not | Strict latency budgets |
| top-k = 5-10 | Strong recall on most corpora | Modest | Typically peaks here | Common production default |
| top-k = 20-50 | Recall plateau | Notable token cost | Starts declining without rerank | Retrieval pool for reranker |
| top-k = 100+ | Marginal recall gain | High cost and latency | Often worse than top-10 | Only as a reranker candidate pool |
Real products, models, and research that use this idea.
- Anthropic Claude Opus 4.7 and GPT-5.5 both still show measurable lost-in-the-middle effects at long contexts, so retrieving 50 chunks into a 200K-context prompt does not eliminate position sensitivity.
- Production RAG stacks built on LlamaIndex or LangChain typically retrieve top-50 then rerank to top-5 with Cohere Rerank or BGE-reranker before the LLM call.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set top-k empirically when you have a labeled eval set?
Sweep k across {3, 5, 10, 20, 50}. Measure retrieval recall@k AND end to end faithfulness (RAGAS or a judge model). Pick the k that maximizes faithfulness, not recall. Beyond the knee, faithfulness drops while latency climbs.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Pushing top-k to a high number to maximize recall while ignoring the three downstream costs: token cost and latency, lost-in-the-middle attention sag, and precision dilution from marginal chunks.
60 second bullets to scan on the way to the call.
Why retrieval recall saturates while costs keep climbing as top-k grows
The lost-in-the-middle attention pattern and its impact on long prompts
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.