A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?
A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?
Measure retrieval (context precision, context recall) and generation (faithfulness, answer relevance) separately using a golden set with known source passages, so you can diagnose failures at the right layer.
Imagine a research assistant who goes to the library and brings back books, then writes a report. If the report is bad, you need to know: did they bring back the wrong books, or did they read the right books and write a bad report? A RAG eval checks both steps separately. First you grade the book selection. Then you grade the writing. You prepare a set of test questions where you already know which books contain the answers, so you can score each step on its own.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The PM's question sounds simple: is retrieval helping? Answering it well requires decomposing the RAG pipeline into its two stages and measuring each independently. A single end to end accuracy number hides whether failures come from the retriever missing the right chunks or the generator ignoring good chunks.
The eval design that answers this question has four components: two retrieval metrics (context precision and context recall), two generation metrics (faithfulness and answer relevance), a golden set that enables all four, and a no-retrieval baseline that isolates retrieval's contribution. The rest of this walkthrough develops each component.
Retrieval metrics: context precision and context recall
Context precision measures the signal to noise ratio in the retrieved chunks. If the retriever returns 10 chunks and only 3 are relevant, precision is 30%. High precision means the generator's context window is clean; low precision means the generator has to filter noise, which increases hallucination risk.
Context recall measures coverage. If 5 passages in the corpus answer the question and the retriever found 3, recall is 60%. Low recall means the generator is missing information it needs, even if the chunks it did retrieve are clean.
Both metrics require a golden set where each question is paired with the specific passages that answer it. Without labeled passages, you cannot compute recall (you do not know what was missed) and precision degrades to a heuristic. Building this golden set is the most expensive part of the eval, and it is the part teams most often skip.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from ragas import evaluate
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy,
)
from datasets import Dataset
# Golden set: questions + known source passages + reference answers
golden = Dataset.from_dict({
"question": questions,
"answer": rag_pipeline_answers,
"contexts": retrieved_chunks, # list of lists
"ground_truth": reference_answers,
})
result = evaluate(
golden,
metrics=[context_precision, context_recall,
faithfulness, answer_relevancy],
)
print(result) # per-metric scores + per-question breakdownReal products, models, and research that use this idea.
- RAGAS (Retrieval Augmented Generation Assessment) is the standard open-source framework for computing context precision, context recall, faithfulness, and answer relevance on RAG pipelines.
- Braintrust and DeepEval both ship RAG-specific evaluation modules that automate the four-metric decomposition against golden sets.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour golden set has 100 questions but the document corpus has 50,000 pages. How do you keep the golden set representative as the corpus grows?
Version the golden set alongside the corpus. When new document categories are added, extend the golden set with questions from those categories. Run a coverage check: if any document cluster has no golden-set questions, the eval has a blind spot for that region of the corpus.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Measuring only end to end answer quality without separating retrieval from generation, making it impossible to diagnose whether the retriever or the generator caused a failure.
60 second bullets to scan on the way to the call.
Why RAG eval must separate retrieval quality from generation quality
What context precision and context recall measure and how they differ
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.