How do you determine if a 2% eval score drop between model versions is statistically significant?
Your eval pipeline shows a 2 percentage point drop in quality score after a model update. How do you determine whether this is a true regression or random variance? Describe the statistical tools available and what you need to run them.
A 2pp drop is meaningless without a confidence interval on the delta. Use paired tests (bootstrap CI or McNemar), pin the judge so its noise does not masquerade as a regression, and gate per slice.
Imagine weighing yourself before and after a diet on a cheap bathroom scale. The scale wobbles by a pound each time you step on it. If you lost two pounds, is that real or just the scale jittering? You would weigh yourself many times and see whether the average drop is bigger than the wobble. An eval score works the same way. The 2-point drop is your weight change. Your noisy judge and your small test set are the wobbly scale. To trust the drop, you measure the wobble (a confidence interval), and you weigh the same examples on both model versions so you are comparing like for like. If the drop is clearly bigger than the wobble, it is a real regression. If not, it is just the scale.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Regression CI for LLMs is the gate that decides whether a new model or prompt version is safe to promote. It sounds like ordinary continuous integration, but the signal it watches is a noisy statistical estimate, not a deterministic test result. A code unit test passes or fails the same way every run. An LLM eval score moves run to run because the eval set is finite and the judge is often stochastic.
This question is the core skill: turning a raw 2-percentage-point delta into a defensible promote or block decision. The honest first answer is that 2pp alone tells you nothing. You cannot interpret a point estimate without bounding its noise.
The deep dive walks the full machinery: separating the noise sources, pinning the judge so its variance does not impersonate a regression, choosing a paired test, reading a confidence interval on the delta, gating per slice with a multiple-comparison correction, and trading fast feedback against thoroughness in the CI design. The thread running through all of it is a single mental shift: stop treating the eval score as a number, and start treating it as a distribution. Every decision below follows from asking how wide that distribution is and whether two of them genuinely separate.
Why a raw delta is uninterpretable
Your eval score is a statistic computed from a sample of examples graded by a noisy instrument. Like any statistic it has a standard error. The reported number is a point estimate of true model quality, and the true value sits somewhere inside an interval around that point.
A 2pp drop between two versions is the difference of two such estimates. The right question is not whether the number went down but whether the drop is large relative to the combined noise in both estimates. If your eval set is small, or the judge is stochastic, the run to run noise floor can easily be several percentage points. In that regime a 2pp drop is indistinguishable from the version not changing at all.
The concept that makes the delta interpretable is the confidence interval on the difference. Compute the per-version scores, compute the delta, and put an interval around that delta. If the 95 percent CI excludes zero, you have evidence of a real change. If it straddles zero, you cannot yet claim a regression. Gating directly on the raw delta, with no interval, is the single most common mistake and it produces a CI gate that flakes on noise.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust both report per example pairwise deltas with confidence intervals so a regression gate fires only when the interval excludes zero.
- Promptfoo runs eval suites in CI and surfaces per test pass or fail diffs against a baseline, the paired data McNemar needs.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set the regression threshold so the gate is neither too flaky nor too permissive?
Estimate the null distribution by re-running the unchanged baseline against itself a few times and measuring the delta CI width. Set the gate to fire only when the observed CI excludes a delta larger than that empirical noise floor, trading false-alarm rate against missed-regression rate explicitly.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Eyeballing a raw 2pp delta and gating on it directly, with no confidence interval, no paired comparison, and a stochastic judge whose own run to run wobble can produce that 2pp out of pure noise.
60 second bullets to scan on the way to the call.
Why a raw delta is meaningless without a confidence interval
Bootstrap CI on the per example difference and reading whether it excludes zero
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.