Your RAG system's faithfulness metric just dropped from 0.91 to 0.74 while context recall stayed flat at 0.85. What's the most likely pipeline layer to investigate first?

RAG eval splits quality into separate axes so you can tell *which part* of the pipeline broke. Two of them matter here. **Context recall** asks whether retrieval pulled in the chunks that contain the answer. Faithfulness asks whether the generated answer is actually supported by those chunks. In this scenario, faithfulness fell sharply (0.91 to 0.74) but context recall stayed flat at 0.85. Read that literally: retrieval is still finding the right material, but the model's answer no longer sticks to it. That rules out the retriever, the embedding model, and the index, because all of those would have dragged recall down too. The change has to be in a later stage: how the chunks are assembled into the prompt, or the model doing the generating. So the first thing to investigate is the augmentation layer or the LLM. In practice that means checking for a recent prompt edit that weakened the 'answer only from context' instruction, a change in chunk ordering, or a model swap. The metric decomposition told you where to look before you opened a single log.

The key skill here is **error attribution** from metric decomposition. RAG eval frameworks deliberately separate retrieval quality from generation quality so a regression points at a layer instead of leaving you guessing. Context recall measures retrieval: did the fetched chunks contain the ground-truth answer? Faithfulness measures generation: is every claim in the answer supported by the supplied context? They are independent by design. Here faithfulness dropped 0.17 points while recall held flat. Flat recall means the retriever is still surfacing the right chunks. The embedding model, index, and search layer are all exonerated: a fault there would have pulled recall down with it. The breakdown is therefore in how those (correct) chunks get used: the augmentation layer or the LLM. The usual culprits, in order of likelihood, are three. A system-prompt edit that softened the grounding directive is first. A chunk-ordering or context-assembly change that buried the relevant text is next, followed by a model swap that altered grounding behavior. You'd confirm by diffing recent deploys against the metric timestamp. The wrong instinct is to blame the retriever; that's exactly the failure the two-axis decomposition exists to prevent.