Your team is tuning top-k for the retriever in a production RAG system. What is the strongest reason NOT to just push top-k to its highest possible value?

Top-k is the number of chunks the retriever returns for each query. You might think bigger is always safer, more chunks means a higher chance the right one is included. In practice, three forces push back. First, every extra chunk is more input tokens for the LLM. That means more cost and more latency. At small scale this is noise. At ten million queries a day it adds up fast. Second, modern LLMs do not pay equal attention to every position in their input. They attend strongly to the start and end of the prompt and weakly to the middle. This is known as **lost-in-the-middle**. A chunk that lands in the middle of a 40-chunk prompt is almost invisible to the model, so retrieving it was wasted effort. Third, low-relevance chunks dilute precision. The model has to decide which chunks to trust. Adding 30 marginal chunks alongside 5 relevant ones gives the model more chances to pick up a wrong fact. The empirical pattern is a saturation curve. Recall climbs sharply at low top-k, then flattens around 5-15. Faithfulness peaks there and often declines as you keep pushing.

Top-k tuning is a three-way tradeoff, not a recall maximization problem. **Direct costs scale linearly with k.** Each extra chunk is more input tokens. That means more dollars per call and more latency. Cross-encoder rerankers also pay this cost: reranking 100 candidates costs roughly 10x reranking 10. At enterprise volume the line is material. **The lost-in-the-middle effect** is the indirect cost. LLMs attend most to the beginning and end of long contexts and least to the middle. Empirically, an LLM asked to find a fact buried at position 20 of 40 chunks finds it much less reliably than the same fact at position 1 or position 40. Retrieving chunks that will land in the attention sag zone is wasted spend. **Precision dilution** is the most subtle. A marginal relevance chunk does not just take up space; it actively competes with the gold chunks for the model's attention. The model has to decide which chunks to trust, and more low-quality candidates raise the probability that it picks up a confidently wrong fact from one of them. The empirical shape is a saturation curve. Faithfulness rises with k until somewhere around 5-15 (corpus dependent), then plateaus or declines. Production tuning is to find that knee on labeled eval data, size to the latency budget, and use a reranker to recover precision when k must be larger.