**A bug in a RAG or agent system can sit at three layers**: the retrieval layer (wrong chunks fetched), the context layer (right chunks fetched but assembled poorly), or the generation layer (right context but the model ignored it). Pure output metrics like BLEU or answer accuracy cannot tell you which layer broke.
The metrics that probe the context layer are:
- **Context precision**, of the chunks assembled into the prompt, what fraction was actually relevant to the question? Low precision means retrieval is wide but noisy.
- **Context recall**, of the facts the gold answer requires, what fraction is present in the assembled context? Low recall means retrieval missed something.
- **Faithfulness**, of the claims in the model's answer, what fraction is grounded in the assembled context? Low faithfulness means the model is hallucinating despite having ground truth in front of it.
- **Gold chunk position**, where in the prompt did the most relevant chunk land? This matters because of position bias (the lost in the middle effect).
The two that distract are BLEU (a text overlap score on the output, insensitive to context) and TTFT (a latency metric, says nothing about quality). Both are fine in their own categories, but neither will surface a broken retrieval pipeline.
**A correct answer with bad retrieval is a liability, not a success.** The reason context layer metrics matter is that they catch the failure modes that pure output metrics cannot see.
The canonical four to remember are the **RAG triad** plus position:
- **Context precision** measures relevance density in the prompt. High precision means the model is not wading through noise.
- **Context recall** measures whether the necessary evidence is present at all. Recall is the metric that catches a retriever silently failing on a hard query.
- **Faithfulness** (also called groundedness) measures whether the answer's claims trace back to the context. Faithfulness catches hallucination even when surface accuracy looks fine, because the model can be right by coincidence.
- **Gold chunk position** correlates with answer quality through the lost in the middle effect. A gold chunk at position 1 typically beats the same chunk at position 8, even with identical chunk content.
**The metrics that look relevant but are not:**
- **BLEU** scores n gram overlap with a reference answer. It rewards surface similarity and penalizes paraphrase. For retrieval grounded answers, two semantically identical responses can score wildly different BLEU. The metric is largely insensitive to whether retrieval was good.
- **Time to first token** is a latency metric. It tells you the model is fast, not that it is right. Some of the worst RAG systems are also the fastest because they skip retrieval entirely.
The production pattern is to log all four context metrics on every call (Ragas, TruLens, and OpenAI Evals all instrument these) and graph them per route. A regression in precision points at a noisy retriever. A regression in recall points at a misconfigured index. A regression in faithfulness points at the model's behavior under that specific context shape.
Quick reference
TL;DR
Context precision, context recall, faithfulness, and gold chunk position localize bugs to the context layer. BLEU and latency say nothing about whether retrieval is broken.
Imagine the cook (the model) made a bad meal. You can taste the meal and say 'it's bad', but that does not tell you whether the cook is bad, or the ingredients were wrong, or the right ingredients were sitting on a shelf and the cook never reached for them. Context quality metrics check the ingredients on the counter. Did the right ones make it from the pantry? Did the cook actually use what was on the counter, or invent things? Was the key ingredient at the front of the counter or hidden behind everything else? End taste alone never answers those questions, and that is why pure output scores miss the bug.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pure generation tasks with no retrieval, creative writing, summarization of a single provided document, code from a spec There is no retrieval layer to evaluate, so the RAG triad collapses. Faithfulness still applies (does the summary stick to the source?) but precision and recall are not meaningful.
Tasks where there is no gold answer or no gold context, open ended exploration, brainstorming Recall requires knowing what was supposed to be in context. Without that ground truth you can only judge faithfulness against the assembled context, not optimality.
When the cost of running an LLM as judge eval per call exceeds the value of the metric in your environment Each judge call is a separate model invocation with its own token bill. For high traffic systems, sample rather than score every call, or run the eval offline against logs.
Quick scan · 2–4 min
Everything important, quickly.
Context layer bugs need context layer metrics. The RAG triad is precision (what fraction of context is relevant), recall (what fraction of needed facts is present), and faithfulness (what fraction of answer claims are grounded). Add gold chunk position to catch lost in the middle effects. BLEU is a surface text overlap score that misses retrieval bugs; TTFT is a latency metric, not a quality one. Each triad regression maps to a specific failing component, which is why these metrics localize bugs that output metrics cannot.
Real-world examples
Real products, models, and research that use this idea.
1Ragas (an open source library) implements precision, recall, and faithfulness as LLM as judge metrics and is the canonical Python tool for RAG evaluation in 2026.
2TruLens and Phoenix (Arize) ship the same triad plus tracing integration so you can see precision and faithfulness alongside each span in production.
3LangSmith and Langfuse both expose evaluator hooks for the triad and run them as continuous offline jobs against gold sets.
4RULER and BABILong are long context evaluation suites specifically designed to surface position bias and lost in the middle effects that the RAG triad alone misses.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you guard against LLM as judge bias when measuring faithfulness?
Sign in or go Pro to see the approach for this follow-up.
QWhat does it mean if precision is high, recall is high, but faithfulness is low?
Sign in or go Pro to see the approach for this follow-up.
QHow would you instrument these metrics in production without doubling your token cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating answer correctness as the only signal. A correct answer with poor retrieval is fragile and you will not see the breakage until the question shifts.