Building a golden dataset for RAG eval from scratch: walk through the steps and name the mistake that wastes the most labeling budget.
Free · 2 AI evals / day
A golden dataset is the ground-truth answer key your eval harness compares model outputs against. Building one well has five steps.
First, sample queries that represent real usage. Pull from production logs if you have them, or synthesize across your known use cases (factoid lookup, multi-hop reasoning, unanswerable queries). The goal is coverage, not volume.
Second, run each query through the retrieval pipeline to get candidate passages. Annotators write answers grounded in those passages, not from memory, because the eval measures what the system actually retrieves.
Third, domain experts write reference answers tagged with the passage IDs that support them. Fourth, a second annotator independently labels at least 20 percent of the set, and you compute inter-annotator agreement (target Cohen kappa above 0.7). Disagreements get adjudicated.
Finally, stratify by query type and difficulty, then freeze a versioned snapshot so future runs stay comparable.
The biggest budget waste is skipping the IAA step. Without it, you label 500 examples and then realize annotators interpreted the rubric differently. Half the labels are inconsistent and the metrics built on them are meaningless. A 50-example pilot catches the problem early.
**Golden-set construction** follows five steps: sample diverse queries from production logs or synthetic generation, run retrieval to get candidate passages, have domain experts write reference answers grounded in those passages, run a second annotator on 20+ percent overlap to compute inter-annotator agreement, then stratify by query type and difficulty and freeze a versioned snapshot.
The step that wastes the most budget when skipped is IAA. Without it, you discover after 500 labels that annotators disagreed on the rubric, making roughly half the labels inconsistent. Downstream metrics computed on noisy labels look stable but are not measuring what you think. A 50-example IAA pilot before full labeling costs a few hours and catches rubric drift before it contaminates the whole set.
Key details that matter in practice: annotators must write answers grounded in the retrieved passages, not from background knowledge, because the eval measures the retrieval pipeline end to end. Each reference answer should include the supporting passage IDs so faithfulness metrics can be computed. The final set should be balanced across query types so no single shape dominates the aggregate score.
Quick reference
TL;DR
Sample queries, retrieve passages, expert-label with IAA, stratify, and freeze. Skipping IAA wastes the most budget because inconsistent labels invalidate downstream metrics.
Imagine building a math answer key for a school test. You write the test questions, solve each one, then hand the answer key to a second teacher to check. If you skip the second teacher and find out later that half your answers were graded on a different rubric, you have to redo the whole thing. The same happens with a golden dataset: if annotators disagree on what counts as a good answer and you only find out after labeling hundreds of examples, you throw away half your work. The fix is cheap: have two people label a small batch first, compare, and agree on the rules before scaling up.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Early prototyping where the retrieval pipeline changes weekly and any golden set would be outdated before labeling finishes. Invest in golden-set construction only after the pipeline architecture stabilizes. Use LLM-as-judge on a handful of manual spot-checks during rapid iteration.
A task where ground truth is inherently subjective (creative writing, open-ended brainstorming) and annotator agreement will always be low. Golden sets assume a right answer exists. For subjective tasks, preference-based evaluation (pairwise comparison, Elo rating) is a better fit than reference-answer matching.
A production system with abundant implicit feedback signals (click-through rate, task completion) that already measure quality at scale. Implicit signals can replace or supplement a golden set for aggregate quality tracking. Reserve golden-set investment for diagnosing specific failure modes the implicit signals cannot isolate.
Quick scan · 2–4 min
Everything important, quickly.
Walk the five steps in order: query sampling, retrieval execution, expert labeling with passage attribution, IAA with a kappa target, stratification and freezing. Name skipping IAA as the costliest mistake and explain the rework math. Cover passage-level attribution for decomposed metrics like Ragas. Close with versioning discipline and quarterly refresh cadence.
Real-world examples
Real products, models, and research that use this idea.
1Ragas, the open-source RAG evaluation framework, decomposes golden-set evaluation into context relevance, faithfulness, and answer correctness, all of which require passage-level attribution in the golden set.
2Anthropic's model evaluations use human-labeled preference datasets with multi-annotator overlap and adjudication to maintain label quality across thousands of examples.
3Google DeepMind's Gemini evaluation suites version their golden sets per release and track lineage so score comparisons across model generations are valid.
4Industry MLOps platforms like Weights and Biases and LangSmith provide golden-set versioning and diff tools so teams can audit what changed between evaluation runs.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour golden set has 200 examples but your RAG pipeline serves 50 different query types. How do you decide which types to over-sample and which to under-sample?
Sign in or go Pro to see the approach for this follow-up.
QYour annotators achieve kappa 0.85 on factoid queries but only 0.55 on multi-hop reasoning queries. What do you do?
Sign in or go Pro to see the approach for this follow-up.
QHow do you keep the golden set from going stale as the product and user behavior evolve?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Skipping inter-annotator agreement and discovering after hundreds of labels that annotators used different rubric interpretations, making half the labels unusable.