Order the steps of an LLM CI/CD eval gate from trigger to decision
- 1Run the golden set through the candidate and compute eval scores
- 2Compare candidate scores to the main branch baseline using a significance test
- 3PR is opened modifying a prompt template or model configuration
- 4Apply the candidate prompt/model change to the eval environment
- 5Block the merge and post a regression report requiring human review
A CI eval gate flows by data dependency: trigger on the PR, apply the change, run the golden set, compare to baseline with a significance test, then block or pass.
Think of a factory adding a new machine to the line. First someone files the request to swap it in. Then they actually install the new machine on a test line. Next they run a known batch of parts through it and measure the results. They line those measurements up against the numbers the old machine produced and check whether the difference is real or just random wobble. Only then does the foreman decide. If quality dropped for real, stop the line and call for review. If it held, let it ship. Each step needs the one before it. You cannot measure a machine you have not installed, and you cannot decide before you have measured.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An eval gate is the LLM equivalent of a unit-test gate in CI. It runs automatically when someone changes a prompt or model, measures whether the change regresses quality on a fixed test set, and blocks the merge if it does. The discipline matters because LLM outputs have no string-match correct answer, so quality has to be measured statistically rather than asserted.
This question asks you to order the five stages from trigger to decision. The order is not a matter of taste. Each stage consumes the artifact produced by the previous one, so the data dependency graph forces the sequence. The deep dive walks each stage, explains why it sits where it does, and covers the senior refinements that separate a gate teams trust from a flaky one they learn to bypass.
Stage 1: the trigger
The gate is event-driven. The trigger is a pull request that modifies a prompt template, a model configuration, a retrieval setting, a tool definition, or any artifact that can change model behavior. This is the natural entry point because CI already runs on PR open, and the eval gate slots in alongside the existing lint and unit-test checks. Reusing the same PR workflow means the eval result lands as just another status check the reviewer already knows how to read.
Scoping the trigger matters. The gate is slow and expensive relative to a unit test, so it should fire only when the PR actually touches behavior-affecting files. Most teams express this as a path filter in the CI config: run the gate when files under the prompts, configs, or retrieval directories change, and skip it otherwise. Wiring it to run on every commit, including pure refactors and documentation edits, wastes judge budget and trains engineers to ignore a check that is usually irrelevant to their change.
The trigger also fixes what counts as the baseline. The PR is branched from a specific main-branch commit, and that commit is the reference point the candidate will eventually be compared against. Anchoring the comparison to the commit the branch forked from keeps regression signals stable even as main moves forward underneath long-lived branches.
The trigger produces one artifact for the next stage: a candidate revision of the prompt or model that needs to be evaluated.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Braintrust both ship CI eval datasets that run on PR and compare a candidate run against a baseline experiment.
- Promptfoo runs as a GitHub Action that scores prompt changes against a fixed test set and fails the check on regression.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhich significance test would you use to compare candidate and baseline eval scores, and why?
For continuous scores use a bootstrap confidence interval on the score delta; for paired pass/fail outcomes on the same items use McNemar. Both account for run to run variance instead of trusting a single mean.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Comparing raw mean scores without a significance test, so normal run to run judge noise gets reported as a regression and blocks merges that were actually fine.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.