Select the metrics that actually measure context quality, not just generation quality

**A bug in a RAG or agent system can sit at three layers**: the retrieval layer (wrong chunks fetched), the context layer (right chunks fetched but assembled poorly), or the generation layer (right context but the model ignored it). Pure output metrics like BLEU or answer accuracy cannot tell you which layer broke. The metrics that probe the context layer are: - **Context precision**, of the chunks assembled into the prompt, what fraction was actually relevant to the question? Low precision means retrieval is wide but noisy. - **Context recall**, of the facts the gold answer requires, what fraction is present in the assembled context? Low recall means retrieval missed something. - **Faithfulness**, of the claims in the model's answer, what fraction is grounded in the assembled context? Low faithfulness means the model is hallucinating despite having ground truth in front of it. - **Gold chunk position**, where in the prompt did the most relevant chunk land? This matters because of position bias (the lost in the middle effect). The two that distract are BLEU (a text overlap score on the output, insensitive to context) and TTFT (a latency metric, says nothing about quality). Both are fine in their own categories, but neither will surface a broken retrieval pipeline.

**A correct answer with bad retrieval is a liability, not a success.** The reason context layer metrics matter is that they catch the failure modes that pure output metrics cannot see. The canonical four to remember are the **RAG triad** plus position: - **Context precision** measures relevance density in the prompt. High precision means the model is not wading through noise. - **Context recall** measures whether the necessary evidence is present at all. Recall is the metric that catches a retriever silently failing on a hard query. - **Faithfulness** (also called groundedness) measures whether the answer's claims trace back to the context. Faithfulness catches hallucination even when surface accuracy looks fine, because the model can be right by coincidence. - **Gold chunk position** correlates with answer quality through the lost in the middle effect. A gold chunk at position 1 typically beats the same chunk at position 8, even with identical chunk content. **The metrics that look relevant but are not:** - **BLEU** scores n gram overlap with a reference answer. It rewards surface similarity and penalizes paraphrase. For retrieval grounded answers, two semantically identical responses can score wildly different BLEU. The metric is largely insensitive to whether retrieval was good. - **Time to first token** is a latency metric. It tells you the model is fast, not that it is right. Some of the worst RAG systems are also the fastest because they skip retrieval entirely. The production pattern is to log all four context metrics on every call (Ragas, TruLens, and OpenAI Evals all instrument these) and graph them per route. A regression in precision points at a noisy retriever. A regression in recall points at a misconfigured index. A regression in faithfulness points at the model's behavior under that specific context shape.