Explain faithfulness vs answer relevance in RAG evaluation with concrete examples

In **RAG** evaluation, the system first retrieves some context documents and then a generator writes an answer using them. Two metrics check very different things about that answer. Faithfulness asks: is every statement in the answer actually supported by the retrieved context? You break the answer into individual claims and check each one against the context. If a claim is not backed by the context, the answer is unfaithful, even if the claim happens to be true in the real world. Answer relevance asks a separate question: does the answer actually address what the user asked? An answer can be perfectly grounded in the context yet completely off-topic. The key insight is that these two dimensions are independent. A high-faithfulness, low-relevance answer copies the context accurately but ignores the real question, like summarising French geography when asked for the capital of France. A high-relevance, low-faithfulness answer nails the question but pulls facts from the model's memory that the context never mentioned, like listing a drug's side effects when the retrieved page only covered dosage. Good RAG needs both to be high at once.

**Faithfulness** measures whether every claim in the generated answer is entailed by the retrieved context. You decompose the answer into atomic claims and check each one against the context. Answer relevance measures something different: does the answer address the user's actual question? These reference points differ. Faithfulness compares the answer to the retrieved context. Relevance compares the answer to the question. Because the reference points differ, the two scores move independently. High faithfulness, low relevance: a user asks for the capital of France. The retrieved context is an article on French geography. The generator writes a faithful summary of that article. Every claim is grounded, so faithfulness is high. But it never states the capital, so relevance is low. High relevance, low faithfulness: a user asks for the side effects of a drug. The retrieved context only covers dosage. The generator answers the side-effects question correctly from its own memory. Relevance is high. But none of those claims appear in the context, so faithfulness is low and hallucination risk is real. The takeaway: report both. Collapsing them into one number hides which half of the pipeline broke.