Your RAG dashboard shows faithfulness consistently above 0.9, but users complain the answers don't help them. Explain what high faithfulness does and does not guarantee, and which evaluation axes you are failing to measure. Name the additional metrics you would add.
Faithfulness only checks the answer is grounded in the retrieved chunks — not that the right chunks were found or that the question was answered. Add context recall and answer relevance.
Imagine a student who only ever copies straight from the textbook, never making anything up. Their answers are always 'faithful' to the book. But if they flipped to the wrong chapter, or answered a question you didn't ask, copying perfectly still leaves you stuck. That's a RAG system scoring high on faithfulness. It proves the model isn't inventing facts — it stuck to its notes. It says nothing about whether it grabbed the right notes, or whether it actually answered your question. You need two more checks: did it find the right page, and did it respond to what you actually asked?
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A green metric that drifts from user happiness is one of the most instructive situations in applied ML, because it forces you to ask what the metric actually measures versus what you assumed it measured. Faithfulness above 0.9 alongside unhappy users is exactly that situation, and it is a favorite interview scenario because the wrong instinct — 'the metric must be miscalibrated' — leads nowhere.
The metric is fine. It is measuring precisely what it claims to: grounding. The gap is that grounding is one stage in a multi-stage pipeline, and a single-stage metric will happily report health while a different stage fails. RAG retrieves, then generates; faithfulness only ever looks at the generation step, and even then only at one property of it.
This deep dive unpacks what faithfulness conditions on, walks the two concrete failure modes that produce the reported symptom, introduces the two axes that bracket it, and shows how reading the three together turns a confusing dashboard into a stage by stage diagnosis. The recurring lesson: any metric that takes an upstream stage as given can look perfect while the system is broken.
What faithfulness actually measures
Faithfulness is a conditional measurement. It takes the retrieved context as a fixed given and asks one question: is every claim in the generated answer supported by — entailed by — that context? Concretely, a judge decomposes the answer into individual statements and checks each against the retrieved chunks. The score is the fraction of statements that the context supports.
Notice what is baked into that definition. The context is the reference. Faithfulness never asks whether the context was the right context, because the context is the thing it is measuring against. It is structurally incapable of catching a retrieval error, the same way a proofreader checking that an essay matches its cited sources cannot tell you the student cited the wrong sources.
This is why faithfulness is the easiest of the three axes to score well on, and why it is dangerous as a sole metric. Two behaviors that should worry you actually inflate it. Summarizing a single retrieved chunk closely tracks that chunk, so it scores high. And abstaining — 'the provided context does not contain that information' — is trivially grounded, because there is nothing to contradict, so it scores high too. A metric that rewards abstention is not a metric you can ship behind alone. Faithfulness earns its place as a hallucination guard, not as a verdict on whether the system helped anyone.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Axis | Question it answers | Stage it isolates |
|---|---|---|
| Context recall | Did retrieval surface the chunks with the answer? | Retriever |
| Faithfulness | Is the answer grounded in the retrieved chunks? | Generation grounding |
| Answer relevance | Does the reply address the user's intent? | Generation usefulness |
Real products, models, and research that use this idea.
- RAGAS scores faithfulness, context precision/recall, and answer relevance as separate axes so a team can localize whether retrieval or generation is the weak link.
- A support bot scoring 0.92 faithfulness while users churn turned out to have low context recall — the reranker was burying the correct policy chunk below the cutoff.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you actually compute context recall, and what ground truth does it need?
You need a labeled set where each query is tagged with the gold chunk(s) or the ground-truth answer. Context recall is then the fraction of answer-relevant statements (or gold chunks) that the retrieved set actually contains. An LLM judge can attribute each statement in the reference answer to whether the retrieved context supports it. Without that ground truth you cannot measure recall — only precision-style proxies.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading high faithfulness as 'the RAG system is working.' A faithful answer can summarize the wrong retrieved chunk, or faithfully say 'the context doesn't contain that' — perfectly grounded, completely useless.
60 second bullets to scan on the way to the call.
State precisely what faithfulness measures and what it conditions on.
Give two distinct high-faithfulness failure modes that still disappoint users.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.