How do you integrate LLM eval into a CI/CD pipeline so regressions block merges?

A **CI/CD eval gate** is the LLM version of a unit-test gate. In normal code, the pipeline runs tests on every pull request and refuses to merge if they fail. For an LLM feature, the equivalent of a test failing is a quality regression: the new prompt or model scores worse on a fixed set of examples than the version already in production. The gate has a few moving parts. A trigger decides when it runs: only on pull requests that touch a prompt template, a model setting, or a system prompt, not on frontend or infrastructure changes. A frozen golden set of 100 to 500 examples lives in the repository so every run grades the exact same questions. An eval run applies the change, scores the golden set with the same judge used in production, and compares against the baseline score from the main branch. The hard part is deciding when a drop is real. Eval scores wobble run to run because judges are not perfectly stable. So you do not block on any drop, you block only when the drop is bigger than that normal wobble and statistically significant. When the gate fails it blocks the merge, posts a report of which examples got worse, and lets a human override after review.

**A CI/CD eval gate** treats quality like a test suite. It runs on pull requests that touch a prompt template, model config, or system prompt. Frontend and infra changes skip it. The gate needs a frozen golden set. That is a version-controlled set of 100 to 500 examples checked into the repo. Freezing it is non-negotiable. If you refresh the set each run, a score change could mean the model regressed or the questions got harder. You cannot tell which. The run applies the candidate change. It scores the golden set with the same judge and rubric as production. It compares to the baseline score stored from the main branch. The regression test is statistical, not a raw delta. Judges wobble a few points between runs. So you use McNemar's test or a bootstrap confidence interval. You block only when the drop clears your threshold and is significant. Set the threshold relative to noise. Measure run to run variance first. Start the gate at roughly twice that variance, then tighten as the pipeline stabilises. Use stricter per-slice thresholds for high-stakes query types. On failure the gate blocks the merge, posts which examples and slices regressed, and requires a human override. After merge, a canary eval in staging acts as a second gate before full rollout.