Describe how to integrate LLM evaluation into a CI/CD pipeline as a quality gate. What components are needed, how is the regression threshold set, and what happens when the gate fails?
Trigger on prompt or model changes, score a frozen golden set against a stored baseline, and block the merge only when the drop beats both run to run noise and a significance test.
Think of a factory line that bottles soda. Before any recipe tweak ships, you pour a fixed set of 200 sample bottles and have the same taste-tester rate them. You keep last week's scores written down. If the new recipe scores clearly worse, the line stops and a person has to look before it restarts. The trick is the taste-tester is a little moody day to day, so you only stop the line if the new batch is worse by more than that normal moodiness, and worse on enough bottles that it is not just a fluke. You never swap the sample bottles between tests, otherwise you cannot tell whether the recipe got worse or the bottles just got harder. After the line restarts you also watch a few real customers quietly before serving everyone.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A CI/CD eval gate is what turns LLM quality from a thing you check by hand into a thing the pipeline enforces. The promise is the same as a unit-test gate: a change that makes the product worse cannot merge silently. The difficulty is that the signal is fundamentally different. A code test returns a deterministic pass or fail. An eval score is a noisy measurement from an instrument, the judge, that has its own variance and its own biases.
The job of the gate is therefore not to declare quality good or bad in absolute terms. It is to answer one narrow question on every pull request: did this specific change make the product measurably and significantly worse than the version already on the main branch? Everything in the design serves that comparison. The golden set fixes what gets measured, the baseline artifact fixes the reference point, and the significance test decides whether the difference is real.
This question is hard because a naive gate fails in both directions. Block on any score drop and judge noise will randomly veto clean pull requests until the team disables the gate. Block on nothing and real quality loss merges unnoticed. The deep dive walks the full architecture, the statistical core that separates signal from noise, how to handle a flaky judge, how to keep the golden set alive without breaking comparability, and how to tune the strictness dial against shipping velocity.
Trigger scope: what actually runs the gate
The gate should run only on changes that can plausibly move model output. That means prompt templates, system prompts, model version bumps, decoding parameters like temperature or top-p, and retrieval configuration for a RAG system. A frontend-only diff, a logging tweak, or an infrastructure change should skip the gate entirely.
The reason is cost and signal. A full judge eval over a few hundred examples costs real money and real wall-clock time. Running it on inert diffs slows every pull request and trains the team to ignore the check. A gate that takes twenty minutes on a CSS tweak is a gate engineers learn to route around. Scoping the trigger by changed paths keeps the gate fast where it matters and invisible where it does not.
In practice this is a path filter in the CI config plus a label or convention for the rare case where an infra change does affect output, for example a library upgrade that changes tokenization. Those get opted in manually.
There is a second, subtler benefit. By scoping the trigger tightly you also keep the comparison clean. When the gate only fires on output-affecting diffs, a failure is almost always attributable to that diff, not to some unrelated change riding along in the same branch. That tight attribution is what makes the per-PR signal trustworthy enough to block on.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Promptfoo runs as a GitHub Action that scores a checked-in eval set per PR and fails the check on a configured regression threshold.
- LangSmith and Braintrust both store baseline scores as artifacts and surface per-example regression diffs against the main branch run.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep a flaky judge from blocking a clean PR or passing a real regression?
Pin temperature to zero and a fixed model version, re-run borderline gates a few times, and require the regression to persist across runs before blocking. Track judge agreement with human labels so you know its variance budget.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Gating on a raw score drop with no significance test. Judge noise alone produces a few points of wobble, so a naive threshold either blocks clean PRs or waves through real regressions.
60 second bullets to scan on the way to the call.
Trigger scope and which diffs should skip the gate
Why the golden set must be frozen and version controlled
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.