Your eval shows Model B scores 82.1% vs Model A at 80.5% on 200 test cases. Your manager wants to ship Model B immediately. What question should you ask first?

When Model B scores 82.1% and Model A scores 80.5% on the same 200 test cases, the natural reaction is to ship Model B. But with only 200 examples, the confidence interval around each score is roughly plus or minus 2.5 to 3 percentage points. That means the true performance of Model B could be anywhere from about 79% to 85%, and Model A could be anywhere from about 78% to 83%. The ranges overlap heavily. The right first question is: is this 1.6-point gap statistically significant? A paired bootstrap test resamples the 200 per-example outcomes thousands of times and computes a confidence interval for the difference. If the 95% interval includes zero, the observed gap could be noise. The other options miss the point. Training-set performance tells you nothing about generalization. An absolute quality bar is useful but does not validate the comparison claim. A harder subset would lower scores, not raise them, and is irrelevant to whether the gap is real.

**Statistical significance** is the first question because 200 test cases produce wide confidence intervals. A rough binomial approximation gives a standard error of about 2.7 points for an 82% score on 200 cases, so the 95% CI for each model spans roughly plus or minus 5.4 points. A 1.6-point gap sits well inside that noise band. The paired bootstrap is the standard test here. Resample the 200 per-example outcome pairs (both models on the same example) with replacement, compute the score difference for each resample, and check whether the 95% CI for the difference excludes zero. A McNemar test is the parametric alternative for binary outcomes. If neither test rejects the null, the improvement is not reliably demonstrated. Shipping on a non-significant gap risks a false-positive model swap: you incur the migration cost, and the next evaluation on fresh data may show Model A was actually better. The cost of a significance test is minutes of compute. The cost of a false-positive ship is a rollback and lost trust in the eval process.