How do you determine if a 2% eval score drop between model versions is statistically significant?

A **regression CI** gate is the eval check that runs before you promote a new model version. The hard part is that eval scores are noisy, so a small drop might be a real regression or might just be random wobble. A 2-percentage-point drop on its own tells you nothing. You need to know the noise floor: how much the score moves run to run even when nothing changed. The tool for that is a **confidence interval** on the difference between the two versions. If the interval around the drop excludes zero, the drop is likely real. If it straddles zero, you cannot conclude a regression yet. To build that interval you need paired data, meaning the same eval examples scored under both the old and the new model. Pairing removes example to example difficulty as a source of noise, so the comparison is much tighter. Two standard tests work on paired data: bootstrap resampling for continuous scores, and McNemar's test for pass or fail outcomes. Two preconditions matter. The eval set must be large enough to have statistical power, often a hundred or more examples per slice. And the judge must be pinned, run at temperature zero with a fixed model version, or its own randomness leaks into the measured difference and you end up testing noise.

**A 2pp drop is ambiguous until you bound the noise.** First, fix the measurement instrument. Pin the judge model version and set temperature to zero so its randomness does not show up as a fake regression. Then test the delta on paired data, meaning the same examples scored under both versions. Two paired tests cover most cases. For continuous or rubric scores, bootstrap the per example difference. Resample the eval set with replacement around a thousand times, recompute the mean delta each time, and report the 95 percent CI. If that interval excludes zero, the regression is real. For binary pass or fail metrics, use McNemar's test on the off diagonal counts, examples the new model fails but the old passes versus the reverse. Pairing matters because it cancels per example difficulty, which is the largest noise source. Power matters too. With small slices you need a wide effect or many examples, typically a hundred or more per slice. Then gate per slice, not just on the global mean, since a regression on one query type can hide inside a flat average. If you cannot pin the judge, run it several times per example and average to shrink its variance before testing.