Which test is most appropriate for determining if a binary metric LLM eval drop is statistically significant?
McNemar's test is the right tool: the data is paired (same examples, two models) and binary (pass/fail), and it tests only the off-diagonal disagreement cells.
Imagine two graders mark the same 200 exam papers as pass or fail, and you want to know whether one grader is genuinely harsher or just unlucky. You ignore the papers both graders agree on, because those tell you nothing about a difference. You look only at the papers where the two disagree: one says pass, the other says fail. If almost all the disagreements lean one way, the harsher grader is really different. If they split evenly, the gap is probably noise. That is exactly what McNemar's test does for two model versions scored on the same fixed eval set: it counts the disagreements and asks whether their lopsidedness is too big to be chance.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Regression testing an LLM feature in CI means running a fixed eval set on every change and asking a sharp question: did this commit make the product measurably worse? When the metric is binary, each example either passes or fails a rubric, the naive instinct is to watch the aggregate pass rate and panic when it dips. That instinct is wrong, because eval scores wobble run to run, and a two-point dip on a 200-example set is often pure noise.
The right move is a significance test, and the right test depends on the structure of the data. The defining feature of a regression eval is that the SAME examples run under both the old and the new model. That makes the outcomes paired. This question is really a check that you recognize paired binary data and reach for the test built for it: McNemar's. The deep dive walks the table, the statistic, why each distractor fails, and the production wrinkles (gate design, judge nondeterminism, multiplicity) that separate a textbook answer from a senior one.
Why the data is paired, not independent
A regression eval scores one fixed golden set under two model versions. Example 42 is graded under the old model and under the new model. The two outcomes for example 42 are linked: hard examples tend to fail under both, easy ones pass under both. That linkage is the definition of paired data.
This is exactly why a two-sample t-test is the wrong tool. A t-test for two independent groups assumes the two samples are drawn separately, with no example-level correspondence. Feeding it paired data throws away the pairing, treats correlated observations as independent, and produces a mis-specified, usually under-powered, test. The pairing is information, and the correct test exploits it rather than discarding it.
The practical tell in an interview: the moment you hear 'same eval set, two models,' your mental model should jump to paired tests. For binary outcomes that is McNemar's. For continuous scores it is a paired t-test or a bootstrap over per-example differences.
The payoff of pairing is statistical power. Because hard and easy examples affect both models the same way, that shared difficulty cancels out when you focus on the per-example difference. You are left measuring only the model to model effect, with the example to example variance subtracted away. An unpaired test cannot do this subtraction, so it has to fight through the full spread of example difficulty and needs a far larger sample to reach the same confidence.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Promptfoo CI gates surface per-case pass/fail, so a paired McNemar table over the old vs new prompt or model falls out directly.
- DeepEval and LangSmith run a fixed golden set on every commit, exactly the paired regression setup McNemar's test was built for.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you turn a McNemar p-value into an actual CI merge gate that does not spam false alarms?
Combine significance at a chosen alpha with a minimum effect size (absolute pass-rate drop). Block only when the regression is both significant and large enough to matter, and correct alpha across sliced metrics.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for a two-sample t-test on a fixed eval set. The same examples run under both models, so the outcomes are paired, not independent. A paired test is required.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.