Order the steps of an LLM CI/CD eval gate from trigger to decision
- 1Run the golden set through the candidate and compute eval scores
- 2Compare candidate scores to the main-branch baseline using a significance test
- 3PR is opened modifying a prompt template or model configuration
- 4Apply the candidate prompt/model change to the eval environment
- 5Block the merge and post a regression report requiring human review
A CI eval gate flows by data dependency: trigger on the PR, apply the change, run the golden set, compare to baseline with a significance test, then block or pass.
Think of a factory adding a new machine to the line. First someone files the request to swap it in. Then they actually install the new machine on a test line. Next they run a known batch of parts through it and measure the results. They line those measurements up against the numbers the old machine produced and check whether the difference is real or just random wobble. Only then does the foreman decide. If quality dropped for real, stop the line and call for review. If it held, let it ship. Each step needs the one before it. You cannot measure a machine you have not installed, and you cannot decide before you have measured.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: the five stages and why data dependency fixes their order, plus the senior refinements on significance testing, golden set hygiene, reproducibility, and keeping the gate fast.
Real products, models, and research that use this idea.
- LangSmith and Braintrust both ship CI eval datasets that run on PR and compare a candidate run against a baseline experiment.
- Promptfoo runs as a GitHub Action that scores prompt changes against a fixed test set and fails the check on regression.
- OpenAI Evals is used in CI to gate prompt and model swaps against a curated golden set before they reach production.
- DeepEval integrates with pytest so prompt regressions surface as failing test cases inside the normal CI run.
- Teams pin a Claude Opus 4.7 or GPT-5.5 judge at temperature zero in the gate so the baseline stays reproducible across runs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhich significance test would you use to compare candidate and baseline eval scores, and why?
QHow do you keep the gate fast enough that engineers do not bypass it?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Comparing raw mean scores without a significance test, so normal run to run judge noise gets reported as a regression and blocks merges that were actually fine.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.