Drag each answer to line up with its matching prompt
Recall@k
Reciprocal of the rank of the first relevant result, averaged over queries
Hit rate
Whether at least one relevant document made it into the top-k (binary per query)
MRR
Generation metric: whether the answer's claims are entailed by the retrieved context
NDCG
Fraction of relevant documents that appear anywhere in the top-k results
Faithfulness (LLM-as-judge)
Graded relevance scored with a position discount, normalized against the ideal ranking
Hit rate and recall@k measure presence in the top-k, MRR measures the rank of the first hit, NDCG scores graded ordering with a position discount, and faithfulness judges the generated answer, not retrieval.
Imagine a librarian fetching books for your question. Hit rate asks: did at least one useful book land in the stack? Recall@k asks: of all the useful books, what fraction made the stack? MRR asks: how near the top of the stack was the first useful one — top of the pile is best. NDCG is the picky judge who also cares how useful each book is and rewards putting the most useful ones highest. Faithfulness is a different judge entirely: it ignores the fetching and instead checks whether your final spoken answer actually matches what those books said.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RAG evaluation confuses people because a single number — 'answer quality' — hides at least two independent things: whether the right context was fetched, and whether the model used it honestly. The five items in this matching exercise are the standard vocabulary for pulling those apart.
This deep dive defines each metric by the exact quantity it computes, shows the formulas for the ranking metrics so the differences are precise rather than vibes, and explains the diagnostic logic that makes the retrieval versus generation split the most useful distinction in the whole RAG evaluation toolkit.
Two stages, two kinds of metric
A RAG system has two stages that can fail independently. The retriever fetches a top-k set of chunks for the query. The generator reads those chunks and writes an answer. A failure in either looks identical from the outside — a bad answer — so you need metrics that isolate each stage.
Four of the five items measure the retriever. They are computed against a labelled set of query to relevant document pairs: for each query, a human or a strong model marks which documents are genuinely relevant. With those labels you can score any ranked result list the retriever produces, without ever looking at the generated text.
The fifth item, faithfulness, measures the generator. It ignores the ranking entirely and asks whether the answer the model produced is supported by the context it was given.
This is the boundary the whole question tests. If you put faithfulness in the retrieval bucket, you lose the ability to say 'the right context arrived but the model ignored it.' Keeping the two buckets separate is what lets you point at the failing stage instead of guessing. Everything below refines the four retrieval metrics, which differ in what aspect of the ranked list they reward.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS computes faithfulness, answer relevance, and context precision/recall as a standard RAG evaluation suite.
- BEIR and MTEB benchmark dense retrievers primarily with NDCG@10 across diverse retrieval tasks.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is hit rate often the most operationally relevant retrieval metric for RAG specifically?
Many RAG questions can be answered from a single good chunk, so what matters is whether at least one relevant chunk reached the context, which is exactly hit rate. Recall@k and NDCG matter more when answers require synthesizing several documents, where missing any of them degrades the answer.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Lumping faithfulness in with the ranking metrics. Faithfulness judges the generated answer against the context; the other four judge whether and where the retriever surfaced relevant documents.
60 second bullets to scan on the way to the call.
Separate the four retrieval metrics from the one generation metric
Define hit rate as binary presence in the top-k
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.