A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?

Attempt it

Free · 2 AI evals / day

Quick reference

TL;DR

Measure retrieval (context precision, context recall) and generation (faithfulness, answer relevance) separately using a golden set with known source passages, so you can diagnose failures at the right layer.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a research assistant who goes to the library and brings back books, then writes a report. If the report is bad, you need to know: did they bring back the wrong books, or did they read the right books and write a bad report? A RAG eval checks both steps separately. First you grade the book selection. Then you grade the writing. You prepare a set of test questions where you already know which books contain the answers, so you can score each step on its own.

Key concepts

RAG Evaluation Faithfulness Context Precision

Quick scan · 2–4 min

Everything important, quickly.

Open with the two-layer framing: retrieval and generation fail independently and need independent metrics. Walk the four metrics (context precision, context recall, faithfulness, answer relevance). Describe golden-set construction from production queries. Explain the no-retrieval baseline comparison. Close with production monitoring: running the eval on a schedule to catch drift as the document corpus changes.

Code example

python

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import Dataset

# Golden set: questions + known source passages + reference answers
golden = Dataset.from_dict({
    "question": questions,
    "answer": rag_pipeline_answers,
    "contexts": retrieved_chunks,       # list of lists
    "ground_truth": reference_answers,
})

result = evaluate(
    golden,
    metrics=[context_precision, context_recall,
             faithfulness, answer_relevancy],
)
print(result)  # per-metric scores + per-question breakdown

Real-world examples

Real products, models, and research that use this idea.

RAGAS (Retrieval Augmented Generation Assessment) is the standard open-source framework for computing context precision, context recall, faithfulness, and answer relevance on RAG pipelines.
Braintrust and DeepEval both ship RAG-specific evaluation modules that automate the four-metric decomposition against golden sets.
Anthropic's enterprise RAG deployments use faithfulness scoring as a production guardrail, flagging answers where claim-level support from retrieved context drops below a threshold.
Promptfoo supports RAG evaluation workflows with configurable retrieval and generation metrics, integrated into CI pipelines.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour golden set has 100 questions but the document corpus has 50,000 pages. How do you keep the golden set representative as the corpus grows?

Sign in or go Pro to see the approach for this follow-up.

QFaithfulness scoring flags a claim as unsupported, but a human reviewer disagrees. How do you handle judge disagreements?

Sign in or go Pro to see the approach for this follow-up.

QYour RAG pipeline retrieves 10 chunks but the LLM context window fits 5. How does this affect the eval design?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.