Name the metrics you would use to evaluate the retriever component on its own, separate from end to end answer quality, and explain why measuring retrieval in isolation matters for debugging a RAG system.
Score the retriever alone with recall@k, hit rate, MRR, and NDCG on labelled query-chunk pairs; isolating retrieval tells you whether to fix the retriever or the generator instead of tuning the wrong stage.
Imagine a cook who keeps serving bad dishes. Is it because the right ingredients never arrived from the pantry, or because the cook ruined good ingredients? You will waste days if you retrain the cook when the real problem is the pantry. Retrieval metrics check the pantry: did the right ingredients show up, and were they near the front of the shelf? If they did and the dish is still bad, the cook is at fault. Measuring the retriever on its own tells you which of the two to fix, so you stop tweaking recipes when the real bug is an empty shelf.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The single most expensive mistake in RAG debugging is optimizing the stage that is not broken. A team sees wrong answers, assumes the prompt is at fault, and burns weeks tuning instructions while the real problem is that the relevant chunk never made the top-k.
Retrieval metrics exist to prevent exactly that. By scoring the retriever on its own, against ground-truth relevance, you turn a vague 'the answers are bad' into a precise 'retrieval recall is 0.4, so the right context is not arriving.' This deep dive covers the metrics you compute, how to build the labelled set they require, and the binary-search debugging workflow that the isolation unlocks — with a worked example showing the two failure modes diverging.
The metrics and what each isolates
All four retriever metrics share a property: they score a ranked list of chunks against a known set of relevant chunks, never touching the generated answer. That is what makes them retriever-only.
Hit rate is binary per query: 1 if any relevant chunk is in the top-k, else 0. It answers 'did the system find anything useful at all?' For RAG questions answerable from one chunk, this is often the metric closest to user-visible quality.
Recall@k is the fraction of all relevant chunks that made the top-k. For a query with several relevant chunks, hit rate can be 1 while recall@k is 0.2, which exposes a system that finds one chunk but misses the rest — fatal for synthesis questions.
MRR averages the reciprocal rank of the first relevant chunk:
It rewards surfacing the right chunk early, which matters when you truncate context hard and only the top few survive the budget.
NDCG uses graded relevance with a logarithmic position discount, normalized against the ideal ordering, so it captures not just presence but how well the ranking orders chunks by degree of relevance. You reach for it when relevance is a matter of degree rather than yes or no.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS pairs context recall and precision with faithfulness so teams can localize a bad answer to retrieval or generation.
- BEIR and MTEB rank embedding models by NDCG@10, which teams use to pick a retriever before building the generation stage.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build the labelled query and relevant chunk set these metrics depend on?
Start from real or synthetic queries, then label relevance: either human annotators mark which chunks answer each query, or you bootstrap by having a strong LLM generate questions from known chunks so the source chunk is the ground-truth relevant one. Keep a held-out set, audit a sample of labels, and refresh it as the corpus changes.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Tuning the generation prompt to fix bad answers without first checking retrieval. If the right chunk never reached the context, no prompt change can recover it.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.