Design a RAG evaluation harness, what do you measure and how?
Design a RAG evaluation harness, what metrics, what data, what process?
Measure faithfulness, answer relevance, context precision, and context recall on a fixed eval set; run on every deploy; validate the LLM judge against human labels.
Imagine grading a student who's taking an open-book exam. You need four scores. Did they actually use the book and not make stuff up? (faithfulness). Did they answer the question that was asked? (answer relevance). Was the book they were given useful? (context precision). Did the book have enough information at all? (context recall). To grade fairly, you need a fixed set of test questions with known good books and known correct answers. You run the same questions every time you change the student, the book, or the curriculum, and you watch the scores. If they drop, something broke. That's a RAG eval harness, automated grading on a fixed test set, run on every change.
Detailed answer & concept explanation~5 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: 4 metrics + per-stage decomposition + eval set construction + LLM-judge biases + human validation cadence + CI integration.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
# Each row: question, generated answer, retrieved chunks, ground-truth answer + context
dataset = Dataset.from_dict({
'question': [q for q in eval_queries],
'answer': [a for a in generated_answers],
'contexts': [chunks for chunks in retrieved_contexts],
'ground_truth': [gt for gt in ground_truth_answers],
})
result = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# {'faithfulness': 0.87, 'answer_relevancy': 0.91, 'context_precision': 0.78, 'context_recall': 0.82}Real products, models, and research that use this idea.
- RAGAS is the open-source framework that codified the 4-metric decomposition and is widely used in 2026 production RAG stacks.
- Anthropic's internal Claude.ai evals run a per-deploy eval harness over a curated set with human-labeled baselines.
- TruLens and DeepEval are alternative open-source frameworks shipping the same 4-metric pattern with different judge prompts.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle the case where your LLM judge is from the same family as your generator?
QYour eval set scores are stable, but production thumbs-down rate is rising. What do you investigate?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Measuring only end to end answer quality. Separate metrics for retrieval and generation tell you which stage broke when something regresses.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.