Order the layers you'd investigate when debugging a RAG faithfulness regression that just shipped to production, from most likely to least likely cause.
- 1Run an isolated retrieval quality check (same eval set, fixed model and prompt) to see if context recall actually dropped.
- 2Verify the LLM model version hasn't changed in config (API endpoint, model name, deployment id).
- 3Diff the augmentation prompt template against the last known good version (system prompt directives, chunk ordering policy, citation rules).
- 4Check whether the user query distribution changed (input drift) by comparing recent query patterns to last week's.
Order debugging hypotheses by base rate, not by what's easiest to blame: check the prompt diff first, then model version, then run an isolated retrieval eval, and treat query drift as the last resort.
Imagine the office printer suddenly prints garbage. You don't start by suspecting cosmic rays. You first check what changed most recently and most easily: did someone load the wrong paper or swap a setting? Only after the obvious culprits do you investigate slow, weird causes like a failing internal part. Debugging a RAG system that suddenly gives ungrounded answers works the same way. The thing engineers touch most often is the prompt template, which ships to everyone the instant it merges. Next is the model version, which lives in config and can silently change. Then you check retrieval quality, but that needs a careful repeatable test. The user's questions slowly changing is real, but it's slow and the hardest to confirm, so you save it for last.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A faithfulness regression is one of the most common and most dreaded RAG incidents. Faithfulness measures whether the generated answer's claims are actually supported by the retrieved context. When it regresses, the system starts confidently asserting things the chunks never said, exactly the hallucination RAG was supposed to prevent, and it ships to real users.
The interview is not really asking you to recite four layers. It's probing whether you debug by evidence and probability or by gut feeling. The right answer orders hypotheses by base rate of change multiplied by speed to confirm, and the rest of this dive explains why each layer lands where it does and how to run the one experiment that actually isolates retrieval.
What 'faithfulness' means and why a regression is scary
Faithfulness is the fraction of claims in the answer that are entailed by the retrieved context. It is distinct from answer relevance (does the answer address the question) and from context recall (did retrieval find the right material). A faithfulness regression specifically means the generation step started drifting away from its evidence.
This is scary because it is silent. The system still returns fluent, confident answers; nothing throws an error. The only way you noticed is an eval metric or a user complaint. That silence is exactly why a disciplined triage order matters, you cannot eyeball your way to the cause across millions of requests, so you need a runbook that spends investigation time where the probability mass is.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Layer | Base-rate of change | Cost to confirm | Triage rank |
|---|---|---|---|
| Prompt template | High, edited weekly, ships at 100% on merge | Trivial, git diff | 1st |
| Model version | Medium, can change in a config deploy | Cheap, config diff | 2nd |
| Retrieval quality | Medium, but needs reproducible eval | Expensive, replay fixed eval set | 3rd |
| Query drift | Low and slow, a trend not a step | Slowest, compare distributions over time | 4th |
Real products, models, and research that use this idea.
- A Perplexity-style search grounded chat: a prompt-template edit that reorders citations ships instantly to all users and breaks grounding before retrieval is ever suspected.
- An enterprise support bot on Azure OpenAI where a deploy silently repins the model snapshot, shifting faithfulness overnight with no retrieval change.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a CI/CD gate that catches a faithfulness regression before it reaches production?
Run RAGAS faithfulness and context-recall on a frozen eval set in the pipeline; block merge on a relative drop beyond threshold; snapshot prompt and model version per run for diffing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Debugging by suspicion instead of base rate. Engineers blame the LLM or retrieval first because they're mysterious, while the prompt diff, the highest probability cause, sits unread.
60 second bullets to scan on the way to the call.
What a faithfulness regression is versus an answer relevance regression
Why base rate of change drives triage order, not perceived mystery
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.