Pick the strongest signal that your post-rerank top-k is too large

When you tune top-k in a RAG system, you want a measurement that tells you whether the model is being helped or hurt by the retrieved chunks. The cleanest signal comes from two standard RAG-eval metrics moving in opposite directions. **Faithfulness** measures whether the model's answer is grounded in the retrieved chunks, does the answer only make claims that are supported by what was retrieved? **Answer-relevance** measures whether the answer actually addresses the user's question. When top-k is too large, faithfulness can stay high or even go up (the model is still using the chunks, just more of them) while answer-relevance drops (the extra chunks are dragging the answer toward tangentially related topics). That divergence is the fingerprint of distractor dilution. The other options are weaker signals. Longer answers and higher time to first token can come from many causes including model changes or prompt edits. The model citing a low-rerank chunk is sometimes correct, the rerank order itself can be wrong, and the model picking a lower-ranked but better chunk is a feature, not a bug.

**Rising faithfulness with falling answer-relevance** is the diagnostic fingerprint of an over-generous post-rerank top-k. Faithfulness asks: does every claim in the answer trace back to a retrieved chunk? Answer-relevance asks: does the answer actually address what the user asked? Standard RAG-eval libraries like RAGAs, DeepEval, and Ragas score both per query. When you raise top-k past the useful threshold, two things happen at once. First, the model has more chunks to ground in, so faithfulness either holds or rises slightly, every claim still has a citation. Second, those extra chunks are weak matches to the actual question; they pull the model toward adjacent topics and away from the user's intent. Answer-relevance drops as a result. That divergence is hard to fake and hard to explain by any other cause. The distractors do not need to be wrong. They can be perfectly correct documents that happen to discuss a different facet of the topic. The model dutifully integrates them, faithfulness stays clean, and the answer slowly stops being about what the user asked. The other signals are too noisy to act on. Longer answers can come from a chatty model. Higher TTFT can come from larger input. A low-rerank citation can be the rerank being wrong rather than top-k being wrong. The two-metric divergence is the clean lever.