Your team is tuning top-k for the retriever in a production RAG system. What is the strongest reason NOT to just push top-k to its highest possible value?

Attempt it

Quick reference

TL;DR

Higher top-k costs more tokens and latency, then hurts faithfulness through lost-in-the-middle attention sag and precision dilution. Recall saturates around 5-15 while costs keep climbing.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine asking a friend a question and handing them a stack of fifty printed pages they have to read before answering. Three of those pages contain the answer; the other forty seven are vaguely related. Your friend now has to find the three useful pages, ignore the rest, and keep their thinking sharp. Most people would do worse than if you had handed them just five pages that included the three good ones. They would take longer, get distracted, and sometimes pick a wrong fact from the noise. RAG works the same way. More retrieved chunks does not mean better answers past a point. Recall plateaus, latency climbs, and the model's attention spreads too thin across mostly irrelevant content.

Key concepts

RAG system design Attention Embeddings

Quick scan · 2–4 min

Everything important, quickly.

3 min: enumerate the three pressures (cost, lost-in-the-middle, precision dilution), explain why recall saturates while faithfulness peaks then drops, and describe the retrieve wide rerank narrow production pattern.

Side-by-side

top-k size	Recall	Latency / cost	Faithfulness	Use pattern
top-k = 3	Often misses borderline cases	Cheapest	High when retrieval is good, brittle when not	Strict latency budgets
top-k = 5-10	Strong recall on most corpora	Modest	Typically peaks here	Common production default
top-k = 20-50	Recall plateau	Notable token cost	Starts declining without rerank	Retrieval pool for reranker
top-k = 100+	Marginal recall gain	High cost and latency	Often worse than top-10	Only as a reranker candidate pool

Real-world examples

Real products, models, and research that use this idea.

Anthropic Claude Opus 4.7 and GPT-5.5 both still show measurable lost-in-the-middle effects at long contexts, so retrieving 50 chunks into a 200K-context prompt does not eliminate position sensitivity.
Production RAG stacks built on LlamaIndex or LangChain typically retrieve top-50 then rerank to top-5 with Cohere Rerank or BGE-reranker before the LLM call.
Pinecone, Weaviate, and Qdrant all expose top-k as a query parameter; the senior pattern is large retrieval k, small prompt k, mediated by a reranker.
RAGAS measures context-precision and faithfulness directly, which is how teams quantify the precision dilution cost of high prompt-k empirically.
Voyage rerank-2 and Jina Reranker v2 are the 2026 references for the rerank step that decouples retrieval depth from prompt depth.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you set top-k empirically when you have a labeled eval set?

Sign in or go Pro to see the approach for this follow-up.

QHow does adding a reranker change the optimal top-k?

Sign in or go Pro to see the approach for this follow-up.

QWalk me through how lost-in-the-middle changes your prompt assembly when k is large.

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.