Your eval shows Model B scores 82.1% vs Model A at 80.5% on 200 test cases. Your manager wants to ship Model B immediately. What question should you ask first? · Zenaique

Attempt it

Quick reference

TL;DR

Ask whether the 1.6pp gap is statistically significant. With 200 test cases, the confidence interval may be wider than the observed difference.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine flipping two coins 200 times each. One lands heads 82 times and the other 80. You would not bet money that the first coin is truly heavier on the heads side, because the difference is so small it could easily be luck. The same logic applies to model scores. With only 200 test cases, a 1.6-point gap might just be noise. Before shipping, you need a statistical test to check whether the gap is real or could vanish if you tested on a different 200 examples.

Key concepts

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Start with the coin-flip intuition: 200 examples, 1.6-point gap, wide CI. Walk through the paired bootstrap mechanics (resample pairs, compute difference, build CI). Contrast paired vs unpaired tests. Name McNemar as the parametric alternative for binary outcomes. Discuss the false-positive model swap risk. Close with the distinction between statistical and practical significance, and name optional stopping as the pitfall when chasing a p-value.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's Claude model evaluations use paired bootstrap confidence intervals on per-example scores to validate that a new model version is genuinely better before promoting it to production.
Google DeepMind's Gemini evaluation reports include confidence intervals alongside headline scores to distinguish real improvements from measurement noise.
The HELM benchmark from Stanford CRFM reports bootstrap confidence intervals for every model score, making it explicit when two models are statistically indistinguishable.
Production A/B testing platforms at Netflix and Spotify require a minimum sample size and a significance threshold before declaring a winner, applying the same logic to ML model comparisons.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour bootstrap CI for the difference is [minus 0.8, plus 4.0]. The gap is not significant at 95%. Your manager says to run on 500 more examples. Is that a valid approach?

Sign in or go Pro to see the approach for this follow-up.

QYou have 200 examples but the eval metric is a continuous 1 to 5 judge score, not binary pass/fail. Does the same bootstrap approach work?

Sign in or go Pro to see the approach for this follow-up.

QHow many test cases do you need for a 1.6-point difference to be significant at 95% confidence?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating any positive score difference as a real improvement without checking whether the gap exceeds the confidence interval for the sample size.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.