Explain faithfulness vs answer relevance in RAG evaluation with concrete examples
Define faithfulness and answer relevance in the context of RAG evaluation (RAGAS framework). Give one concrete example of a response that is high on faithfulness but low on relevance, and one that is high on relevance but low on faithfulness.
Faithfulness checks every claim is entailed by the retrieved context; relevance checks the answer addresses the question. They measure against different reference points and move independently.
Imagine an open-book exam. Faithfulness asks: did the student write only things that were actually in the textbook they were handed? If they invented a fact that is not on the page, faithfulness drops, even if the fact happens to be true. Answer relevance asks a different question: did the student actually answer what was asked? A student can copy a perfectly accurate paragraph straight from the book (high faithfulness) yet completely miss the question (low relevance). Or they can nail exactly what was asked from memory (high relevance) while writing things that are nowhere in the book they were given (low faithfulness). The two graders look at different things, so a good answer has to please both at once.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Faithfulness and answer relevance are the two headline generation-side metrics in RAG evaluation, and the single most common interview confusion is treating them as the same thing. They are not. They measure the generated answer against two completely different reference points, and a strong candidate can name those reference points instantly.
Faithfulness compares the answer to the retrieved context: is every claim in the answer actually supported by what was retrieved? Answer relevance compares the answer to the user's question: does the answer actually address what was asked? Because the reference points differ, the two scores move independently, and that independence is exactly what makes them diagnostically useful.
This deep dive walks the precise definitions, the claim-level procedure the LLM judge uses to compute faithfulness, the formula, the two canonical examples, the distinction from factual accuracy, and the failure modes that bite in production. The throughline is that faithfulness is mechanical and decomposable, which is exactly why it is the most reliable automatic signal in a RAG harness and the one an interviewer will expect you to compute by hand.
Faithfulness: claim-level entailment from context
Faithfulness asks one question: is every statement in the generated answer entailed by the retrieved context? Crucially, it is not asking whether the statements are true in the real world. A claim that is factually correct but absent from the retrieved passages is unfaithful. Faithfulness rewards grounding, not truth.
The LLM judge computes it in two stages. First, it decomposes the generated answer into a set of atomic claims, where each claim is a single self-contained statement. An answer like 'Paris is the capital and has 2 million residents' becomes two claims. Second, for each claim, the judge runs a natural language inference style check: can this claim be inferred from the retrieved context? Each claim gets a supported or not-supported verdict, and a claim counts as supported if at least one retrieved chunk entails it.
This decomposition is what separates faithfulness from a holistic 'does this look grounded' rating. A fluent answer can bury one hallucinated claim among five grounded ones; a holistic score smooths over it, while per-claim scoring surfaces it. The per-claim resolution is also what makes the metric actionable: you get the exact list of unsupported statements, not just a number, so you can show an engineer which sentence the generator invented.
The granularity is the metric's main weak spot. How finely you split compound sentences changes the denominator, so two runs with slightly different decompositions can disagree on the score even when the answer and context are identical. This is why the decomposition prompt is pinned and validated, not left to the judge's whim.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS computes faithfulness by prompting an LLM judge to extract atomic claims, then entailment-check each against the retrieved context.
- TruLens ships a groundedness feedback function that mirrors RAGAS faithfulness, scoring per-claim support from source passages.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the LLM judge actually compute the faithfulness score step by step?
Walk the two-stage prompt: first decompose the answer into atomic claims, then run a natural-language-inference check of each claim against the context. Score is supported claims divided by total claims. Mention temperature zero and human-label validation.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating faithfulness and relevance as one quality score. A grounded answer can miss the question, and an on-point answer can be ungrounded, so collapsing them hides the real failure.
60 second bullets to scan on the way to the call.
The two reference points: context for faithfulness, question for relevance
Claim decomposition then per-claim entailment as the faithfulness procedure
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.