Building a golden dataset for RAG eval from scratch: walk through the steps and name the mistake that wastes the most labeling budget.

A golden dataset is the ground-truth answer key your eval harness compares model outputs against. Building one well has five steps. First, sample queries that represent real usage. Pull from production logs if you have them, or synthesize across your known use cases (factoid lookup, multi-hop reasoning, unanswerable queries). The goal is coverage, not volume. Second, run each query through the retrieval pipeline to get candidate passages. Annotators write answers grounded in those passages, not from memory, because the eval measures what the system actually retrieves. Third, domain experts write reference answers tagged with the passage IDs that support them. Fourth, a second annotator independently labels at least 20 percent of the set, and you compute inter-annotator agreement (target Cohen kappa above 0.7). Disagreements get adjudicated. Finally, stratify by query type and difficulty, then freeze a versioned snapshot so future runs stay comparable. The biggest budget waste is skipping the IAA step. Without it, you label 500 examples and then realize annotators interpreted the rubric differently. Half the labels are inconsistent and the metrics built on them are meaningless. A 50-example pilot catches the problem early.

**Golden-set construction** follows five steps: sample diverse queries from production logs or synthetic generation, run retrieval to get candidate passages, have domain experts write reference answers grounded in those passages, run a second annotator on 20+ percent overlap to compute inter-annotator agreement, then stratify by query type and difficulty and freeze a versioned snapshot. The step that wastes the most budget when skipped is IAA. Without it, you discover after 500 labels that annotators disagreed on the rubric, making roughly half the labels inconsistent. Downstream metrics computed on noisy labels look stable but are not measuring what you think. A 50-example IAA pilot before full labeling costs a few hours and catches rubric drift before it contaminates the whole set. Key details that matter in practice: annotators must write answers grounded in the retrieved passages, not from background knowledge, because the eval measures the retrieval pipeline end to end. Each reference answer should include the supporting passage IDs so faithfulness metrics can be computed. The final set should be balanced across query types so no single shape dominates the aggregate score.