Complete the claim: the two axis decomposition of RAG eval that diagnoses which layer failed.
Faithfulness measures whether the answer is grounded in the retrieved chunks; context recall measures whether retrieval found the chunks at all. They isolate generation vs retrieval failures.
Picture an open book exam. Two things can go wrong. First, the student writes an answer that contradicts the book, or makes up facts the book never stated. That is a writing problem. Second, the student turned to the wrong pages, so the book in front of them never contained the answer at all. That is a finding problem. RAG evaluation splits quality into exactly these two questions. Faithfulness asks whether the written answer stays true to the pages that were open. Context recall asks whether the right pages were ever found. Keeping them separate matters: if you only measure the final answer, a wrong answer tells you nothing about which step failed. Splitting the score tells you whether to fix the retriever or the generator.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Ask a candidate "how do you evaluate a RAG system" and the weak answer is "we check if the answers are good." The strong answer recognizes that RAG is two systems bolted together (a retriever that fetches evidence and a generator that writes from it) and that a single quality number can't tell you which one is broken. That insight is the entire reason RAG evaluation is decomposed into orthogonal axes.
The two foundational axes are faithfulness and context recall. Faithfulness lives on the answer side and grades the generator: is what the model wrote actually supported by the chunks it was given? Context recall lives on the context side and grades the retriever: did the chunks even contain the information a correct answer needs? This deep dive walks through each axis, why they must stay independent, the four-quadrant diagnosis that makes them actionable, and the production caveats around measuring them at scale.
Faithfulness: grading the generator
Faithfulness answers one narrow question: are the claims in the generated answer grounded in, and not contradicted by, the retrieved context? It is purely an answer vs context comparison. It never looks at the outside world or any ground-truth label.
The usual computation decomposes the answer into atomic claims, then checks each claim for entailment against the retrieved chunks, either with a natural language inference model or an LLM judge. The score is the fraction of claims that the context supports:
Low faithfulness has two flavors worth distinguishing. The first is fabrication: the model adds a claim the context never made, the classic hallucination. The second is contradiction: the context says one thing and the answer asserts the opposite. Both drop the score, and both point at the generation stage rather than retrieval.
The critical subtlety is what faithfulness does not measure. It does not tell you the answer is correct. If the retriever surfaces a stale or wrong document and the model dutifully repeats it, the answer is perfectly faithful and completely wrong. Faithfulness is a measure of honesty relative to the evidence, not of truth. It also says nothing about whether the answer addresses the user's actual question; a faithful but off-topic answer scores high here and only fails the separate answer relevance metric. This is exactly why faithfulness must be paired with a context-side metric rather than read in isolation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Faithfulness | Context recall | Diagnosis | Where to fix |
|---|---|---|---|
| High | High | Working as intended | Nothing: this is the target |
| High | Low | Honest but ignorant (retrieval failure) | Chunking, embeddings, hybrid search, k |
| Low | High | Ignored good context (generation failure) | Prompt, context ordering, model |
| Low | Low | Broken on both ends | Retriever and generator both |
Real products, models, and research that use this idea.
- RAGAS computes faithfulness by extracting claims from the answer and verifying each against retrieved context, alongside context recall and context precision.
- TruLens scores the RAG Triad (context relevance, groundedness (faithfulness), and answer relevance) to localize failures by pipeline stage.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow is faithfulness actually computed when there's no labeled ground truth for every query?
Decompose the answer into atomic claims, then use an LLM judge or NLI model to check each claim is entailed by the retrieved context; score = supported claims / total claims. Calibrate the judge on a human labeled subset.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Collapsing both axes into one end to end accuracy score. A single number hides whether the retriever missed the chunk or the generator ignored it, so you can't tell which layer to fix.
60 second bullets to scan on the way to the call.
What faithfulness measures and which pipeline layer it grades
What context recall measures and how it differs from context precision
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.