Explain how a golden set regression suite fits into a prompt change CI workflow
A golden-set regression suite is 50-200 hand-curated items the candidate prompt must clear on every PR before merge. CI scores the run, compares against a stored baseline, and fails the build on regressions.
Think of a flight simulator before a pilot is cleared to fly the real plane. The simulator does not test every possible flight; it tests a small set of scenarios that, if the pilot can handle them, give the airline enough confidence to let them go. A golden-set regression suite is the same idea for a prompt. You hand-pick a few dozen examples that you know the system has to get right, and any time someone wants to change the prompt, those examples have to be run again. If the new version handles them as well as the old one did, the change ships. If it slips on any of the must-pass cases, the build fails and the pull request stays red until someone fixes it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The golden-set regression suite is the most-misunderstood piece of an LLM eval stack. Teams either skip it entirely and rely on dataset runs that take an hour to finish, or they try to make the golden set carry every job and grow it to thousands of items until CI takes longer than the rest of the build.
The right framing is to treat it as one tier in a layered strategy. Its only job is to be the fastest possible pre-merge gate against regressions on cases the team has explicitly decided must keep working. Everything else, from broad coverage to drift detection, is somebody else's job.
What the suite contains and how it is shaped
A working golden set has 50 to 200 items. Each item is a structured record: input, expected behavior, evaluation method, and a tag for whether it is must-pass or soft.
Must-pass items represent non-negotiables. The system must refuse this request. The system must not hallucinate this customer's name. The system must compute this arithmetic correctly. A must-pass failure is a binary CI fail regardless of aggregate score. Pick 10-30 items for this tier.
Soft items contribute to an aggregate score. They cover the broader space of behaviors you want to keep stable, but where you accept a small regression in exchange for a different gain. The aggregate threshold is typically a percentage-point delta (2-5) below the stored baseline.
Evaluation method per item is one of: exact match (for items with a deterministic expected output), regex or schema match (for structured outputs), or judge-scored (for free-text outputs). The judge is itself an LLM prompt that returns a score 0-1 or a categorical label. The judge prompt and the judge model version must be pinned in the suite config; if the judge drifts, baselines stop being meaningful.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse Datasets in 2026 supports 'dataset runs' bound to a prompt version, with a CI-friendly API to fail the build on score regression.
- LangSmith's evaluators run on dataset versions and emit comparison reports between two prompt versions, designed for the PR-gate workflow.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep the golden-set scorer stable when LLM-as-judge models change underneath you?
Pin the judge model and prompt version in the suite config; treat judge upgrades as a separate event that requires re-baselining the entire suite under the new judge before being trusted.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Growing the golden set to thousands of items 'for better coverage'. It stops being golden, becomes slow, and engineers start bypassing it.
60 second bullets to scan on the way to the call.
Golden-set size range and why it stays small
Must-pass subset versus soft aggregate threshold
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.