A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?
A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?
Measure retrieval (context precision, context recall) and generation (faithfulness, answer relevance) separately using a golden set with known source passages, so you can diagnose failures at the right layer.
Imagine a research assistant who goes to the library and brings back books, then writes a report. If the report is bad, you need to know: did they bring back the wrong books, or did they read the right books and write a bad report? A RAG eval checks both steps separately. First you grade the book selection. Then you grade the writing. You prepare a set of test questions where you already know which books contain the answers, so you can score each step on its own.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the two-layer framing: retrieval and generation fail independently and need independent metrics. Walk the four metrics (context precision, context recall, faithfulness, answer relevance). Describe golden-set construction from production queries. Explain the no-retrieval baseline comparison. Close with production monitoring: running the eval on a schedule to catch drift as the document corpus changes.
from ragas import evaluate
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy,
)
from datasets import Dataset
# Golden set: questions + known source passages + reference answers
golden = Dataset.from_dict({
"question": questions,
"answer": rag_pipeline_answers,
"contexts": retrieved_chunks, # list of lists
"ground_truth": reference_answers,
})
result = evaluate(
golden,
metrics=[context_precision, context_recall,
faithfulness, answer_relevancy],
)
print(result) # per-metric scores + per-question breakdownReal products, models, and research that use this idea.
- RAGAS (Retrieval Augmented Generation Assessment) is the standard open-source framework for computing context precision, context recall, faithfulness, and answer relevance on RAG pipelines.
- Braintrust and DeepEval both ship RAG-specific evaluation modules that automate the four-metric decomposition against golden sets.
- Anthropic's enterprise RAG deployments use faithfulness scoring as a production guardrail, flagging answers where claim-level support from retrieved context drops below a threshold.
- Promptfoo supports RAG evaluation workflows with configurable retrieval and generation metrics, integrated into CI pipelines.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour golden set has 100 questions but the document corpus has 50,000 pages. How do you keep the golden set representative as the corpus grows?
QFaithfulness scoring flags a claim as unsupported, but a human reviewer disagrees. How do you handle judge disagreements?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Measuring only end to end answer quality without separating retrieval from generation, making it impossible to diagnose whether the retriever or the generator caused a failure.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.