Why an eval as CI gate needs a significance test, not just a raw score delta

Flashcard·Hard·4.0 · 0·~30s·Asked atGoldman Sachs Qdrant Weaviate

Attempt it

Probably not on that signal alone. With 100 examples and judge-score noise, a 0.03 absolute drop is well inside what a paired-bootstrap or McNemar test will return as not significant at typical alpha levels. Blocking on raw deltas this size produces a high false-positive rate, trains the team to ignore the gate, and discourages legitimate prompt improvements.

The right setup runs both prompts on the same examples (paired design), bootstraps the per-example score difference (5000 resamples is plenty), and blocks only when the 95 percent confidence interval excludes zero in the bad direction. Some teams add a worst-slice rule (block if any tracked slice drops more than a wider threshold) so a small aggregate drop hiding a large per-tenant regression is still caught.

Your answer

An **eval-CI gate** is a CI job that runs your golden set against the new prompt or model and compares the score to the baseline. If the new score is worse, the gate blocks the merge. The naive setup uses a fixed threshold: block if the score drops more than X. The problem is that LLM judge scores are noisy. The same model, the same prompt, the same example, scored twice by the same judge, can produce different numbers. With a small evaluation set (100 examples is typical), the natural noise alone moves the aggregate score by 0.02 to 0.05 between runs. A gate that fires on a 0.03 drop will fire half the time on prompt changes that did not actually regress anything. The team learns that the gate produces false alarms, starts overriding it, and eventually the gate is reduced to a checkbox that nobody trusts. **The fix is a significance test.** Run both the old and new prompts on the same examples (paired design). For each example, compute the score difference. Bootstrap-resample the per-example differences 5000 times and construct a 95 percent confidence interval. Block only if the confidence interval excludes zero in the bad direction. The size of difference required to fire scales with sample size and judge variance, which is the right calibration. A 0.03 drop on 1000 examples might be significant; the same 0.03 drop on 100 examples usually is not.

**The diagnosis.** A 0.03 absolute drop on a 100-example golden set with LLM judge scoring is almost always inside the noise band. Judge variance from the LLM-as-judge itself, combined with the sampling variance of the 100-example set, makes a 95 percent confidence interval on the aggregate score typically plus or minus 0.04 to 0.06. A 0.03 movement does not cross that band. **Why a fixed-delta rule fails.** The delta threshold has no relationship to either the judge's variance or the set size. On a small set it fires constantly on noise; on a large set it misses real regressions that are smaller than the threshold but statistically significant. Either way the gate's precision and recall are wrong, and the team's response is to stop trusting it. **The paired design.** Run both the old prompt and the new prompt on the exact same examples (same input, same retrieved-context snapshot if RAG-backed, same judge model). For each example i, compute d_i = score_new(i) minus score_old(i). The paired design controls for example-level variance, which is the largest source of noise. A non-paired design (run new on its own, compare to historical baseline) discards this control and needs much larger samples for the same power. **The bootstrap.** Resample the d_i with replacement, compute the mean, repeat 5000 times. The empirical distribution of the resampled means gives a 95 percent confidence interval. Block only when the upper bound of the CI is below zero (for higher is better metrics) or the lower bound is above zero (for lower is better metrics). **The worst-slice check.** Aggregate-level significance is necessary but not sufficient. A 0.01 aggregate drop can hide a 0.15 drop on the enterprise tier offset by a 0.05 gain on the free tier. Run the significance test per stratification slice (tenant tier, intent, difficulty) and block on the worst-slice regression even if aggregate is fine.

TL;DR

A 0.03 drop on 100 examples is well inside judge-score noise; use a paired bootstrap or McNemar to gate only on statistically significant differences, with a separate worst-slice rule to catch hidden per-tenant

Memory aid

Easy to grasp

Picture taking your blood pressure three times. The numbers will not be identical even if your health is the same; the cuff and the measurement itself have noise. Now imagine you decide that any 5-point change between two readings means you need new medication. You will swap medications constantly because the readings naturally bounce. The right rule is to look at the trend across many readings and only act when the change is clearly bigger than the normal bounce. LLM eval scores work the same way. A small drop from 0.84 to 0.81 on a small sample is well within the natural bounce; a gate that fires on it teaches the team to ignore the gate, which is the worst possible outcome.

Key concepts

LLM evaluation LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

An eval-CI gate is the central enforcement mechanism that keeps prompt and model changes from regressing production quality. A gate that fires on noise loses team trust within weeks; a gate that misses real regressions loses customer trust within months. The middle path is statistical significance applied with care.

This explanation walks through why a 0.03 drop on 100 examples is usually not a regression, why paired design is the right test architecture, how to implement the bootstrap or McNemar test in practice, and what additional structure (per-slice tests, worst-slice statistic, calibration against historical decisions) keeps the gate honest over time.

The noise floor of an LLM eval gate

An LLM judge scoring 100 examples produces a number with substantial measurement uncertainty. Three sources of noise dominate.

Judge stochasticity. Even at temperature 0, an LLM judge does not always score the same input identically. Long-context judges and reasoning-heavy judges have more variance than short-context single-pass judges. Empirically, the same judge on the same input produces scores that differ by 5 to 15 percent run to run on a 0-to-1 scale.

Sampling variance. The 100 examples are a draw from a wider distribution of possible test cases. The aggregate score is itself a random variable whose standard error is roughly sigma over sqrt(n), where sigma is the per-example score standard deviation. With sigma 0.2 and n 100, the standard error is 0.02 and the 95 percent CI is roughly plus or minus 0.04.

System-under-test stochasticity. If the production model is run at temperature greater than 0, the response itself differs between runs, which the judge scores differently. Temperature 0 controls this for most providers but not all (some retain stochasticity for tie-breaking).

All three combine to make the noise band on a 100-example aggregate score roughly plus or minus 0.04 to 0.06 in absolute units. A 0.03 movement is well inside that band. Acting on it as a regression is acting on noise.

Why paired design dominates

The bootstrap procedure step by step

Per-slice tests, multiple testing, and calibration

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Promptfoo's CI integration supports paired-comparison runs out of the box; the result table shows per-example deltas suitable for bootstrapping.
DeepEval's pytest assertions support paired-design eval and can be wrapped with a bootstrap significance check before the assertion fails.

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through how you would size the golden set so the gate can detect a 0.02 regression with 80 percent power.

Estimate per-example difference standard deviation sigma from a pilot run (often 0.08 to 0.12 for an LLM judge on a 0 to 1 scale). For a paired t-test with alpha 0.05, power 0.80, and effect 0.02, the required sample size is roughly n = 8 * (sigma / effect)^2. With sigma 0.10 that gives n = 200. The honest takeaway is usually: detecting a 0.02 regression reliably requires 200 to 400 examples, not 100.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Setting a fixed delta threshold like 'block if score drops more than 2 points'. The threshold has no relationship to sample size or judge variance, so it produces high false-positive rates on small sets and high false-negative rates on large sets.

60 second bullets to scan on the way to the call.

Typical judge-score variance on a 100-example set and the resulting minimum detectable effect
Why paired design strips example-level variance and gains power over unpaired

Primary sources. Browse if you want the original framing.