Building a golden dataset for RAG eval from scratch: walk through the steps and name the mistake that wastes the most labeling budget.
Sample queries, retrieve passages, expert-label with IAA, stratify, and freeze. Skipping IAA wastes the most budget because inconsistent labels invalidate downstream metrics.
Imagine building a math answer key for a school test. You write the test questions, solve each one, then hand the answer key to a second teacher to check. If you skip the second teacher and find out later that half your answers were graded on a different rubric, you have to redo the whole thing. The same happens with a golden dataset: if annotators disagree on what counts as a good answer and you only find out after labeling hundreds of examples, you throw away half your work. The fix is cheap: have two people label a small batch first, compare, and agree on the rules before scaling up.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A golden dataset is the ground-truth answer key that your evaluation harness uses to score model outputs. Every metric you compute, every model comparison you run, and every ship decision you make rests on the quality of this dataset. Building one well is not glamorous, but building one poorly wastes more engineering time than almost any other eval mistake.
The procedure has five steps: query sampling, retrieval execution, expert labeling, inter-annotator agreement, and stratified freezing. The failure modes line up with the steps you skip. The rest of this explanation walks each step in detail and explains why skipping IAA is the single most expensive mistake a team can make.
Step 1: sampling queries that represent real usage
The golden set is only as good as the queries it contains. If every query is a simple factoid lookup, your eval score will look great even if the system fails catastrophically on multi-hop reasoning or unanswerable queries.
The starting point is production logs. Sample queries stratified by type: factoid, multi-hop, comparison, unanswerable, and any domain-specific categories your product handles. If production logs are not available (pre-launch), synthesize queries across your known use cases and have domain experts review them for realism.
The target is coverage, not volume. Two hundred well-chosen queries that span every important query type are more valuable than two thousand queries that all look the same. Aim for at least 20 examples per query type so per-type metrics are statistically meaningful.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Ragas, the open-source RAG evaluation framework, decomposes golden-set evaluation into context relevance, faithfulness, and answer correctness, all of which require passage-level attribution in the golden set.
- Anthropic's model evaluations use human-labeled preference datasets with multi-annotator overlap and adjudication to maintain label quality across thousands of examples.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour golden set has 200 examples but your RAG pipeline serves 50 different query types. How do you decide which types to over-sample and which to under-sample?
Weight by business impact and failure frequency. A query type that covers 40 percent of production traffic and has a 15 percent failure rate deserves more examples than a type that covers 2 percent of traffic with near-zero failures. Use production error logs and user feedback signals to estimate failure rates, then allocate golden-set budget proportionally to the product of traffic share and failure rate.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping inter-annotator agreement and discovering after hundreds of labels that annotators used different rubric interpretations, making half the labels unusable.
60 second bullets to scan on the way to the call.
Name the five construction steps in order from query sampling to freezing the set.
Explain why annotators must ground answers in retrieved passages, not background knowledge.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.