Drag each answer to line up with its matching prompt
The document containing the answer never appears in the retrieved set
Augmentation and the generator: prompt assembly and how the LLM uses context
The right passage is retrieved but the answer ignores or contradicts it
Retriever and embeddings: chunking, the embedding model, or the index
Retrieved chunks are unrelated to what the user actually meant
Index freshness: incremental indexing, deletion, and re-embedding
The answer is well grounded but reflects an outdated version of the facts
Query understanding: query rewriting, expansion, or routing
RAG failures localize by one triage question: did the right evidence reach the prompt? Missing evidence points upstream (query, retriever, index freshness); wrong use of present evidence points at the generator.
Imagine a student writing an essay using a library. If the essay is wrong, ask where the chain broke. Did the student ask the librarian the wrong question? Did the librarian hand over the wrong books? Did the library not even own the up to date edition? Or did the student get the right book but ignore what it said? Each failure has a different culprit, and you fix the right one by tracing the chain in order. A retrieval system is the same: the question, the search, the freshness of the shelves, and how the writer uses the page are four separate places a mistake can hide.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
When a RAG system gives a bad answer, the instinct is to reach for the most visible knob — usually the prompt or the model. That instinct is wrong more often than it is right, because most RAG failures are not generation failures at all. They are failures somewhere along the chain that runs from the user's query to the final token, and the generator is just the last link that happened to be holding the bag.
The skill an interviewer is probing here is fault localization: given a symptom, can you name the layer to investigate first without flailing? This deep dive lays out the data path, gives you the one triage question that cleaves the failure space in half, and walks each symptom to its layer. It closes with the instrumentation that turns this from a memorized table into a measurable, data-driven practice.
The data path and the one triage question
A RAG query flows through a chain of stages: the raw query is understood and transformed, the transformed query searches an index, the index returns chunks (its contents governed by how fresh the index is), the chunks are assembled into a prompt, and the model generates. A fault can live at any link.
The trick that makes triage fast is to not inspect every link at once. Instead, ask a single binary question: did the chunk that contains the correct answer actually reach the context window?
This question cleaves the entire failure space in two. If the evidence was absent from the prompt, no amount of prompt or model work can help — the generator never had a chance. The fault is upstream, in query understanding, retrieval, or index freshness. If the evidence was present in the prompt and the answer is still wrong, then retrieval did its job and the fault is downstream, in how the prompt was assembled or how the model used it.
Answering this question is cheap if you instrumented for it — you log the retrieved chunk IDs and check whether the gold chunk is among them. It is nearly impossible if you only look at the final answer. That is why teams who skip retrieval logging end up treating every bug as a generation bug and tuning prompts forever.
It helps to picture the failure space as a decision tree. The root is the triage question. The first branch — evidence absent — fans into three leaves, because there are three distinct stages upstream of the prompt that can keep the right chunk out: the query that drove the search, the search machinery itself, and the freshness of what the search was searching. The second branch — evidence present — has its own leaves downstream in how the prompt was built and how the model used it. The reason this framing is worth internalizing is that it tells you which experiments to run in which order. You never inspect a leaf before you have answered the branch above it, because the branch determines which half of the system is even worth instrumenting. A team that opens the prompt template first, before confirming the chunk was retrieved, has skipped the root of the tree and is debugging blind.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS reports context recall and context precision separately from faithfulness, so you can see whether a failure is retrieval-side or generation-side.
- LlamaIndex and LangChain trace integrations log retrieved node IDs and scores per query, letting you assert whether the gold chunk was present.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you instrument a RAG system so this localization is data-driven rather than guesswork?
Log per-query the retrieved chunk IDs and similarity scores, and on an eval set assert whether the gold chunk was in the set. Compute retrieval metrics (recall@k, MRR) independently from generation metrics (faithfulness, answer relevance). When a failure occurs, the two metric families tell you which half of the pipeline broke before you touch anything.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming the LLM for every bad answer when the evidence chunk was never retrieved — a retrieval failure that no amount of prompt tuning will fix.
60 second bullets to scan on the way to the call.
State the master triage question used to localize a RAG failure
Map missing-evidence symptoms to the upstream layers
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.