Your RAG system's faithfulness metric just dropped from 0.91 to 0.74 while context recall stayed flat at 0.85. What's the most likely pipeline layer to investigate first?

Attempt it

Quick reference

TL;DR

Faithfulness fell but context recall held, so retrieval is still finding the right chunks. The bug lives downstream: the augmentation layer or the generating model, not the retriever.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of an open book exam. Context recall asks: did the student turn to the right pages? Faithfulness asks: did the answer actually match what those pages said? Here the student still found the correct pages (recall is flat), but the written answer no longer lines up with them (faithfulness dropped). So the problem is not in finding the book; it's in how the student read and used it. In a RAG system, that means the retriever is fine. Something changed in how the chunks get handed to the model or in the model itself: maybe a teammate edited the prompt that tells it to stick to the sources, maybe the chunks now arrive in a worse order, or maybe someone swapped the underlying model for one that wanders off-source. You look there first, not at the search index.

Key concepts

RAG system design LLM evaluation Prompt engineering

Quick scan · 2–4 min

Everything important, quickly.

3 min: define both metrics by layer, explain the orthogonal attribution logic, give the triage order, and confirm with a deploy diff before touching code.

Side-by-side

Metric	Layer it measures	What a drop implies
Context recall	Retrieval (embeddings, index, search)	Right chunks aren't being fetched
Context precision	Retrieval ranking	Relevant chunks ranked below noise
Faithfulness	Generation (context + model)	Answer drifts off the supplied chunks
Answer relevance	Generation intent	Answer doesn't address the question

Real-world examples

Real products, models, and research that use this idea.

A team using RAGAS in CI sees faithfulness regress after a prompt template edit while context recall holds, and reverts the template.
An Arize Phoenix dashboard flags a faithfulness drop the same day a Claude Opus to Sonnet swap shipped, isolating it to generation.
A LangSmith eval run shows faithfulness falling after a chunk reordering change buried relevant context lower in the prompt.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QFaithfulness AND context recall both dropped together this time. How does your triage change?

Sign in or go Pro to see the approach for this follow-up.

QHow would you confirm the layer before making any change, rather than guessing from the metric alone?

Sign in or go Pro to see the approach for this follow-up.

QFaithfulness scores depend on an LLM-as-judge. How do you trust the metric itself isn't the thing that drifted?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.