Faithfulness scores are high, yet users keep calling the answers unhelpful — what is the eval missing?

Short answer·Medium·4.0 · 0·~3 min·Relevant atAmazon Apple

Attempt it

Your RAG dashboard shows faithfulness consistently above 0.9, but users complain the answers don't actually help them. Explain what high faithfulness does and does not guarantee, and which evaluation axes you are failing to measure. Name the additional metrics you would add.

Free · 2 AI evals / day

**Faithfulness** measures one narrow thing: are the claims in the answer supported by the chunks that were retrieved? A high score means the model is not hallucinating — it stuck to its sources. That is genuinely valuable, but it is only one link in the chain. It does not check two other things. It does not check whether retrieval found the right chunks in the first place. And it does not check whether the answer actually addresses what the user asked. So you can get a perfect faithfulness score on a useless answer. If retrieval pulled the wrong passage, the model can faithfully summarize it — grounded, but off-target. If retrieval found nothing relevant, the model can faithfully say 'the context does not contain that information' — also grounded, also useless. That is why your users are unhappy while the dashboard is green. You are measuring one axis of a three-axis problem. Add two metrics. **Context recall** checks whether retrieval surfaced the chunks that actually contain the answer — this catches retrieval misses. **Answer relevance** checks whether the response addresses the user's real intent, not just whether it is grounded. Together with faithfulness these form the standard RAG evaluation triad.

Faithfulness answers exactly one question: is every claim in the answer supported by the retrieved context? A 0.9 means the model is grounding its output and not hallucinating. That is worth measuring, but it is silent on the two links that surround it. It says nothing about **retrieval quality**: whether the chunks fed to the model were the right ones. And it says nothing about **answer usefulness**: whether the response addresses the user's intent. Both failures are compatible with a perfect faithfulness score. The model can faithfully summarize the wrong passage, scoring high while answering the wrong thing. Or, when retrieval misses, it can faithfully report 'the context does not contain that information' — impeccably grounded, completely unhelpful. Your green dashboard and your unhappy users are both correct; you are measuring one axis of a three-axis problem. The fix is to evaluate the other two axes. Context recall asks whether retrieval surfaced the chunks that contain the answer — the metric that catches retrieval misses, the likely culprit when users say the answer was off-topic. Answer relevance asks whether the response addresses the question actually asked, independent of grounding. RAGAS packages these as the canonical triad: faithfulness, plus context precision and recall, plus answer relevance. The deeper payoff is localization. Because the axes are separable, a failure points at a specific stage. Low context recall with high faithfulness means fix the retriever. High recall and faithfulness but low answer relevance means fix the generation prompt. You stop trusting one number.

TL;DR

Faithfulness only checks the answer is grounded in the retrieved chunks — not that the right chunks were found or that the question was answered. Add context recall and answer relevance.

Memory aid

Easy to grasp

Imagine a student who only ever copies straight from the textbook, never making anything up. Their answers are always 'faithful' to the book. But if they flipped to the wrong chapter, or answered a question you didn't ask, copying perfectly still leaves you stuck. That's a RAG system scoring high on faithfulness. It proves the model isn't inventing facts — it stuck to its notes. It says nothing about whether it grabbed the right notes, or whether it actually answered your question. You need two more checks: did it find the right page, and did it respond to what you actually asked?

Key concepts

RAG LLM Evaluation LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

2–4 min · Everything important, quickly.

45s: state that faithfulness only checks grounding in retrieved context; give the two high-faithfulness failure modes (wrong chunk summarized, faithful abstention); add context recall for retrieval misses and answer relevance for off-question replies; frame the triad as separable axes that localize the failing stage.

Axis	Question it answers	Stage it isolates
Context recall	Did retrieval surface the chunks with the answer?	Retriever
Faithfulness	Is the answer grounded in the retrieved chunks?	Generation grounding
Answer relevance	Does the reply address the user's intent?	Generation usefulness

Real products, models, and research that use this idea.

RAGAS scores faithfulness, context precision/recall, and answer relevance as separate axes so a team can localize whether retrieval or generation is the weak link.
A support bot scoring 0.92 faithfulness while users churn turned out to have low context recall — the reranker was burying the correct policy chunk below the cutoff.
LangSmith and Langfuse evaluation runs let you attach per-axis judge scores to production traces, so faithfulness and answer relevance are tracked side by side.
TruLens's RAG triad dashboard plots context relevance, groundedness (faithfulness), and answer relevance together to prevent exactly this single-metric blind spot.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you actually compute context recall, and what ground truth does it need?

QTwo systems both score 0.9 faithfulness but one frustrates users — how do you tell them apart?

QWhy are most RAG quality problems retrieval problems, not generation problems?

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reading high faithfulness as 'the RAG system is working.' A faithful answer can summarize the wrong retrieved chunk, or faithfully say 'the context doesn't contain that' — perfectly grounded, completely useless.

60-second night-before-the-interview bullets. Scan these on the way to the call.

Primary sources. Skim if you want the original framing.