When you run a **regression eval** in CI, you score the same fixed set of examples under the old model and the new model. Each example gets a pass or fail under each. Then the new model's pass rate drops a couple of points. Is that a real regression or just noise? You need a significance test.
The key fact is that the data is paired. Example 17 is evaluated under both models, so its two pass/fail outcomes are not independent. Tests that assume independent groups, like a two-sample t-test, are the wrong tool here.
McNemar's test is built exactly for this. It lays the results into a 2x2 table of agreement and disagreement. It throws away the cases where both models agree (both pass, both fail) because those carry no information about a difference. It tests only the two disagreement cells: old passes while new fails, versus old fails while new passes.
If the disagreements are lopsided (the new model fails many cases the old one passed, but rarely the reverse), the drop is significant. If they roughly balance, it is noise. That is why McNemar's is the answer, not the t-test, chi-squared goodness of fit, or just eyeballing variance.
**McNemar's test** is correct because the eval data is both paired and binary. The same fixed set runs under the old and new model. So example-level pass/fail outcomes are paired, not independent.
McNemar's builds the 2x2 table of (old, new) pass/fail. It ignores the concordant cells (both pass, both fail). Those carry no signal about a difference. It tests only the discordant cells: `b` (old pass, new fail) and `c` (old fail, new pass). The statistic is:
$$\chi^2 = \frac{(b - c)^2}{b + c}$$
The null hypothesis is that the two disagreement directions are equally likely. A lopsided split means a real change.
The distractors fail for clear reasons. A two-sample t-test assumes independent groups. Here the groups share examples, so it is mis-specified and understates the paired signal. Chi-squared goodness of fit tests whether one distribution matches an expected shape. It is not a paired two-model comparison. Re-running the eval and checking standard deviation estimates noise. It is not a hypothesis test against a baseline.
Quick reference
TL;DR
McNemar's test is the right tool: the data is paired (same examples, two models) and binary (pass/fail), and it tests only the off-diagonal disagreement cells.
Imagine two graders mark the same 200 exam papers as pass or fail, and you want to know whether one grader is genuinely harsher or just unlucky. You ignore the papers both graders agree on, because those tell you nothing about a difference. You look only at the papers where the two disagree: one says pass, the other says fail. If almost all the disagreements lean one way, the harsher grader is really different. If they split evenly, the gap is probably noise. That is exactly what McNemar's test does for two model versions scored on the same fixed eval set: it counts the disagreements and asks whether their lopsidedness is too big to be chance.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Your metric is a continuous score (faithfulness 0 to 1, judge rating 1 to 5), not binary pass or fail McNemar's needs binary outcomes. Use a paired t-test or a bootstrap confidence interval on the per-example score differences instead.
The two models are evaluated on different, non-overlapping example sets The outcomes are no longer paired, so the McNemar assumption breaks. An independent-group test becomes appropriate.
Your discordant-cell counts are tiny, a handful of disagreements on a small set The chi-square approximation is unreliable. Use the exact binomial form of McNemar's or collect a larger eval set.
You are gating on many sliced metrics in one pipeline run One McNemar's per slice inflates false positives. You need a multiple-comparison correction across the family of tests.
Quick scan · 2–4 min
Everything important, quickly.
5 min: why a fixed eval set is paired binary data, the McNemar 2x2 table and statistic, why each distractor is wrong, then the CI gate, judge-nondeterminism, and multiple-comparison wrinkles.
Real-world examples
Real products, models, and research that use this idea.
1Promptfoo CI gates surface per-case pass/fail, so a paired McNemar table over the old vs new prompt or model falls out directly.
2DeepEval and LangSmith run a fixed golden set on every commit, exactly the paired regression setup McNemar's test was built for.
3Teams comparing GPT-5.5 vs Claude Opus 4.7 on a binary correctness rubric use paired tests, not independent-group t-tests, on the shared eval set.
4OpenAI Evals reports per-example pass/fail, letting CI pipelines compute discordant-cell counts between two model versions.
5RAGAS faithfulness thresholds run on a held fixed set per release, where a significant binary drop, not a raw dip, defines a regression.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you turn a McNemar p-value into an actual CI merge gate that does not spam false alarms?
Sign in or go Pro to see the approach for this follow-up.
QYour eval metric comes from an LLM judge at default temperature. How does that affect McNemar's validity?
Sign in or go Pro to see the approach for this follow-up.
QWhen the metric is a continuous score rather than pass/fail, what replaces McNemar's?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reaching for a two-sample t-test on a fixed eval set. The same examples run under both models, so the outcomes are paired, not independent. A paired test is required.