Your eval pipeline shows a 2-percentage-point drop in quality score after a model update. How do you determine whether this is a true regression or random variance? Describe the statistical tools available and what you need to run them.
Free · 2 AI evals / day
A **regression CI** gate is the eval check that runs before you promote a new model version. The hard part is that eval scores are noisy, so a small drop might be a real regression or might just be random wobble.
A 2-percentage-point drop on its own tells you nothing. You need to know the noise floor: how much the score moves run to run even when nothing changed. The tool for that is a **confidence interval** on the difference between the two versions. If the interval around the drop excludes zero, the drop is likely real. If it straddles zero, you cannot conclude a regression yet.
To build that interval you need paired data, meaning the same eval examples scored under both the old and the new model. Pairing removes example to example difficulty as a source of noise, so the comparison is much tighter. Two standard tests work on paired data: bootstrap resampling for continuous scores, and McNemar's test for pass or fail outcomes.
Two preconditions matter. The eval set must be large enough to have statistical power, often a hundred or more examples per slice. And the judge must be pinned, run at temperature zero with a fixed model version, or its own randomness leaks into the measured difference and you end up testing noise.
**A 2pp drop is ambiguous until you bound the noise.** First, fix the measurement instrument. Pin the judge model version and set temperature to zero so its randomness does not show up as a fake regression. Then test the delta on paired data, meaning the same examples scored under both versions.
Two paired tests cover most cases. For continuous or rubric scores, bootstrap the per example difference. Resample the eval set with replacement around a thousand times, recompute the mean delta each time, and report the 95 percent CI. If that interval excludes zero, the regression is real. For binary pass or fail metrics, use McNemar's test on the off diagonal counts, examples the new model fails but the old passes versus the reverse.
Pairing matters because it cancels per example difficulty, which is the largest noise source. Power matters too. With small slices you need a wide effect or many examples, typically a hundred or more per slice. Then gate per slice, not just on the global mean, since a regression on one query type can hide inside a flat average. If you cannot pin the judge, run it several times per example and average to shrink its variance before testing.
Quick reference
TL;DR
A 2pp drop is meaningless without a confidence interval on the delta. Use paired tests (bootstrap CI or McNemar), pin the judge so its noise does not masquerade as a regression, and gate per slice.
Imagine weighing yourself before and after a diet on a cheap bathroom scale. The scale wobbles by a pound each time you step on it. If you lost two pounds, is that real or just the scale jittering? You would weigh yourself many times and see whether the average drop is bigger than the wobble. An eval score works the same way. The 2-point drop is your weight change. Your noisy judge and your small test set are the wobbly scale. To trust the drop, you measure the wobble (a confidence interval), and you weigh the same examples on both model versions so you are comparing like for like. If the drop is clearly bigger than the wobble, it is a real regression. If not, it is just the scale.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Exact-match metrics on a fully deterministic pipeline with a frozen eval set If both runs are deterministic, any score change is a real diff. A significance test on zero variance is unnecessary ceremony.
Tiny eval slices with only a handful of examples per category There is no power to detect anything. Collect more examples first, or the CI will be too wide to ever exclude zero.
Pre-merge fast-feedback checks where minutes matter more than rigor A full thousand-sample bootstrap across all slices is too slow. Run a small pinned smoke slice and defer the heavy gate to nightly.
Open-ended generative quality with no agreed rubric or gold answer The bottleneck is defining the measurement, not its variance. Stabilize the rubric before any significance test means anything.
Quick scan · 2–4 min
Everything important, quickly.
6 min: why a raw delta is uninterpretable, the two noise sources, pinning the judge, paired bootstrap and McNemar, per-slice gating with multiple-comparison correction, effect size versus significance, and fast-feedback versus thorough gates.
Real-world examples
Real products, models, and research that use this idea.
1LangSmith and Braintrust both report per example pairwise deltas with confidence intervals so a regression gate fires only when the interval excludes zero.
2Promptfoo runs eval suites in CI and surfaces per test pass or fail diffs against a baseline, the paired data McNemar needs.
3OpenAI Evals and DeepEval pin the judge model version and temperature zero so judge variance does not leak into the measured delta.
4Chatbot Arena reports Bradley-Terry ratings with bootstrap confidence intervals so overlapping bars signal a tie rather than a real gap.
5Teams shipping Claude Opus 4.7 or GPT-5.5 upgrades run a fast pinned smoke slice per commit plus a full nightly bootstrap gate before promotion.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set the regression threshold so the gate is neither too flaky nor too permissive?
Sign in or go Pro to see the approach for this follow-up.
QWhy does pairing give more statistical power than comparing two independent score means?
Sign in or go Pro to see the approach for this follow-up.
QYou run CI gates on 40 slices. How do you stop noise from tripping at least one every time?
Sign in or go Pro to see the approach for this follow-up.
QYour eval set grew to 50k examples and now every tiny change is significant. What changed and what do you do?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Eyeballing a raw 2pp delta and gating on it directly, with no confidence interval, no paired comparison, and a stochastic judge whose own run to run wobble can produce that 2pp out of pure noise.