Design a regression-CI gate for prompt-template changes that prevents bad prompts from reaching production.

**The gate has five moving parts.** A golden eval set, two or three metrics, a comparison routine, a gate decision, and a failure-surfacing message. The **golden set** is 100 to 500 representative cases of (input, expected-output, rubric). It lives in version control next to the prompts and gets refreshed quarterly with recent production failures. The metrics are picked so they cover different failure modes. Faithfulness (does the output respect the context). Format compliance (does the output parse). Refusal correctness (does the model refuse on out of scope and not refuse on in-scope). Latency and cost get measured too. Single-metric gates miss regressions on the other axes. The **comparison** runs the new prompt against the eval set, computes per-metric means with bootstrap confidence intervals, and compares against the production baseline. The gate blocks merge when a metric drops past a noise floor (3 to 5 points) with non-overlapping CIs, hard-blocks on format failures, and allows a reviewer override for deliberate tradeoffs. The PR comment surfaces per-metric deltas and concrete examples of regressed cases. Generic 'eval failed' messages train people to ignore the gate.

The gate has to do three jobs. Catch real regressions before they ship. Avoid blocking on sampling noise. Surface failures concretely enough that the author can fix the problem in the same loop. Eval set: 100 to 500 cases in version control. Include representative cases for the main flows, edge cases for known failure modes, and recent production failures. Refreshed quarterly. Keep a separate held-out canary of 5 to 10 tricky cases that never feeds into prompt iteration. That canary catches overfitting to the golden set. Metrics: two or three orthogonal axes, not a single accuracy number. Faithfulness via LLM-as-judge for grounded-QA tasks. Format compliance as a binary pass-fail per item. Refusal correctness as a binary on a held-out scope-violation set. Latency p95 and cost per call tracked separately. The minimum viable set is faithfulness plus format compliance plus refusal correctness for most production prompts. Comparison: bootstrap 1000+ resamples for 95% CIs on each metric. The gate compares means against the current production baseline and looks at CI overlap. Per-metric noise floor is measured empirically: run the production prompt five times and the std dev is the floor (typically 2 to 3 points on a 100-point scale). Gate logic: block merge if any metric drops past the noise floor with non-overlapping CIs. Hard-block on format-compliance drops (no noise floor; the schema is binary). Hard-block on p95 latency exceeding budget. Soft-warn on within-noise drops. Allow reviewer override for deliberate tradeoffs (latency-up for quality-up) with a documented marker in the PR. Failure surfacing: PR comment with per-metric deltas, confidence intervals, three concrete regressed examples, and a link to the full eval report. The author needs enough signal to fix the issue without leaving the PR.