Design a RAG evaluation harness, what metrics, what data, what process?
Measure faithfulness, answer relevance, context precision, and context recall on a fixed eval set; run on every deploy; validate the LLM judge against human labels.
Imagine grading a student who's taking an open-book exam. You need four scores. Did they actually use the book and not make stuff up? (faithfulness). Did they answer the question that was asked? (answer relevance). Was the book they were given useful? (context precision). Did the book have enough information at all? (context recall). To grade fairly, you need a fixed set of test questions with known good books and known correct answers. You run the same questions every time you change the student, the book, or the curriculum, and you watch the scores. If they drop, something broke. That's a RAG eval harness, automated grading on a fixed test set, run on every change.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RAG evaluation became its own subfield around 2023-2024 because the obvious approach (have a human read every answer) doesn't scale, and the alternative (ask an LLM to grade) is treacherous if you don't structure it carefully. The RAGAS framework codified what most mature teams were already doing: decompose the pipeline into independent quality axes, score each separately, and validate the automated judge against humans.
A good eval harness answers three questions: did retrieval find the right context, did generation use that context faithfully, and did the answer actually address what was asked. Each question has a metric; each metric needs an eval set; the whole apparatus needs to run on every deploy.
This deep dive walks through the four metrics, the eval-set construction, the LLM-as-judge pitfalls, and the operational pattern that ties offline scores to production reality.
Why decompose into four metrics
End-to-end answer quality is a single number. When it drops, you can't tell why. Was retrieval surfacing junk chunks? Was the model hallucinating despite good chunks? Was the answer addressing the wrong question entirely? Without per-stage metrics, every regression turns into a manual investigation.
The four-metric decomposition fixes this by tying each metric to one pipeline stage.
Faithfulness scores whether the answer's claims are supported by the retrieved context. It catches hallucination at the generation step. If retrieval was perfect but the model freelanced anyway, faithfulness drops while context precision stays high.
Answer relevance scores whether the answer addresses the question. It catches off-topic generation. If the model retrieved correctly and wrote a faithful summary of irrelevant chunks, faithfulness stays high but answer relevance tanks.
Context precision scores how many retrieved chunks were useful. If retrieval returned 10 chunks and only 2 mattered, precision is low. The model can still produce a good answer (filtering the noise itself), but you're wasting tokens.
Context recall scores whether retrieval surfaced the ground-truth context. If the right answer needed information that retrieval missed, recall drops, and the model has no way to recover.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
# Each row: question, generated answer, retrieved chunks, ground-truth answer + context
dataset = Dataset.from_dict({
'question': [q for q in eval_queries],
'answer': [a for a in generated_answers],
'contexts': [chunks for chunks in retrieved_contexts],
'ground_truth': [gt for gt in ground_truth_answers],
})
result = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# {'faithfulness': 0.87, 'answer_relevancy': 0.91, 'context_precision': 0.78, 'context_recall': 0.82}Real products, models, and research that use this idea.
- RAGAS is the open-source framework that codified the 4-metric decomposition and is widely used in 2026 production RAG stacks.
- Anthropic's internal Claude.ai evals run a per-deploy eval harness over a curated set with human-labeled baselines.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle the case where your LLM judge is from the same family as your generator?
Use a cross-family judge (Claude judges GPT, GPT judges Claude) to break self-preference; or ensemble two judges and only act on agreement.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Measuring only end to end answer quality. Separate metrics for retrieval and generation tell you which stage broke when something regresses.
60 second bullets to scan on the way to the call.
The four core metrics and which stage each scores
Why per-stage metrics beat end to end only metrics
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.