How do you design a golden eval set for a prompt that can serve as a regression-CI gate in production?

Short answer·Hard·4.0 · 0·~3 min·Asked atAnthropic·Relevant atDatabricks Humanloop OpenAI

Attempt it

You're a data scientist building the eval harness for a production LLM app. The team needs a golden eval set that's good enough to act as a regression-CI gate. Walk through: how big should it be, what should it contain, how do you score it, how do you keep it from going stale, and what's the bootstrap process when you have zero production traffic.

Free · 2 AI evals / day

A golden eval set is the curated test bench a prompt has to pass before it ships. For a regression CI gate in production, the recipe is fairly standard. On size, aim for 100 to 500 examples. Under fifty cannot distinguish a real regression from sampling noise. Over five hundred starts costing more in maintenance and LLM as judge bills than it gives you back. Most production tasks land at 100 to 300. On composition, use three buckets. Roughly 60% representative cases sampled from real traffic, 25% edge cases that target specific failure modes, 15% recent production failures pulled from logs. Each example is a triple of input, expected output, and a rubric that defines what 'correct' means for that specific case. On scoring, use two or three orthogonal metrics. For a RAG app, that is usually faithfulness (does the answer contradict the context), coverage (does it include the key facts), and format compliance (valid JSON). Compute per metric mean plus a bootstrap 95% confidence interval so you can tell signal from noise. Keep it fresh. Quarterly refresh, hold out 30% as a final test split you never iterate on, version the eval set in git, and add new production failures as they appear. For a zero-traffic bootstrap, synthesize 50 to 100 examples from the product spec, get a domain expert to label 20 to 30 hard ones, deliberately construct edge cases, and ask beta users to flag wins and losses. Grow over time as real traffic arrives. The gate logic on the CI side is simple. The new prompt has to match or beat the current production prompt on every metric within the bootstrap CI. Any metric that regresses meaningfully blocks the merge.

A regression CI eval set for a production prompt is 100 to 500 triples spread across three buckets. It is scored on multiple orthogonal metrics with bootstrap CIs, and split into iteration and held-out test partitions. On size, 100 is the minimum for bootstrap CIs to discriminate. 300 is the sweet spot. 500 is the ceiling before curation cost dominates. On composition, the rough split is 60% representative cases sampled from real traffic and 25% deliberate edge cases. Edge cases cover unusual structure, long or short input, multi language, and malformed payloads. The last 15% is recent production failures from logs. Each example is `(input, expected_output, rubric)`. The rubric is specific enough that a different annotator (or LLM judge) reaches the same call. On scoring, use two or three orthogonal metrics per task. A faithfulness or correctness judge, a coverage or completeness judge, plus a rule based format compliance check. Compute mean and 95% bootstrap CI on each. On hygiene, hold out 30% as a final test split never used during iteration. Refresh quarterly with new production failures. Version the set alongside the prompt in git and tag which prompt was evaluated against which set version. For zero traffic, synthesize from the spec, get an SME to label, generate adversarial edge cases, recruit beta users. Start at 50 to 100 and grow to 200 to 500 as real traffic arrives. The CI rule is straightforward. A new prompt has to be statistically no worse than current on every metric, with at least one metric materially better, before it merges.

Quick reference

TL;DR

A production golden eval set is 100 to 500 triples (input, expected, rubric) across representative + edge + recent-failure buckets, scored on 2 to 3 orthogonal metrics with bootstrap CIs and a held-out test split.

Memory aid

Easy to grasp

Imagine you are a chef and you want to know whether your new recipe is better than the old one. You do not taste it once and decide; you cook it for a panel of 200 carefully chosen tasters. Some are typical customers, some are picky eaters with unusual diets, some are the people who hated your last dish. Each one rates the meal on three things: flavor, presentation, plate cleanliness. You only call the new recipe better if the average rating beats the old one by more than the panel's natural noise. That panel is the golden eval set; the rating rubric is how you score; the picky eaters keep you honest.

Key concepts

Golden eval set LLM regression testing LLM as judge Bootstrap CIs

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: size and composition + scoring with bootstrap CIs + iteration vs test split + quarterly refresh + zero traffic bootstrap + per slice and judge calibration.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's evals documentation and Claude Workbench tools assume this pattern: golden set with rubric per example and LLM as judge scoring per metric.
OpenAI's Evals framework expects (input, expected, rubric) triples and bootstraps significance across prompt variants for the same task.
DSPy and LangSmith both surface eval sets as first class versioned artifacts you reference by ID across compile and evaluate steps.
RAGAS is the canonical open source harness for RAG with built in faithfulness, answer relevancy, and context recall metrics that match this pattern.
Inflection and Perplexity have publicly described golden sets in the few hundred range plus held out test sets as the gate for shipping new prompts.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you decide when a regression in one metric is acceptable if another metric improves a lot?

QHow do you handle eval contamination when the same examples have been used to iterate on the prompt for months?

QWhat is the right ratio of LLM as judge spend to production inference spend?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Building a 20 example eval set and treating it as a CI gate. Twenty examples cannot distinguish a real regression from noise; you need 100 or more before bootstrap CIs are meaningful.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.