Explain why an LLM product should run a held out eval set on every prompt or model change in CI, and what failure this prevents.
Prompt and model changes regress cases you didn't look at, invisibly. A held-out eval in CI turns quality into a blocking number that catches drift before users do.
Imagine tuning a recipe to make one picky friend happy. You add more salt, they love it — but you never check whether everyone else now finds it too salty. A held-out eval set is like keeping a panel of taste-testers who try every version of the recipe. Before you serve the new version to guests, the panel scores it. If the score drops, you don't serve it. That way one fix for one friend can't quietly ruin the dish for the whole table — and 'I think it's better' becomes an actual score you can trust.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every team building on LLMs eventually has the same bad day: someone improves a prompt, ships it, and a week later support is flooded with complaints about a behavior nobody intended to touch. The change worked perfectly on the example the engineer was looking at. It quietly wrecked a dozen others. This question is about the discipline that prevents that day from happening.
The reason it's worth a whole interview question is that the intuition from normal software doesn't transfer. When you edit a typed function, the type checker and your unit tests form a net that catches far-away breakage. A prompt has neither. It's a soft, natural-language controller whose effect spans the entire input distribution, with no compiler to tell you what you just changed.
This deep dive develops three ideas: why prompt and model changes are non-local and therefore dangerous, why a held-out eval set in CI is the regression net that replaces the missing compiler, and why that net is only as strong as its coverage — which makes maintaining the eval set the real ongoing work.
Why a prompt change is never a local change
Start with the mental model that gets people in trouble. They picture a prompt edit like editing a function body: a contained change with contained effects. So they test the one case they care about, see it fixed, and ship.
But a prompt isn't a function with narrow inputs. It's a global controller sitting in front of the model's behavior on every possible input. Adding a sentence like "always be concise" to fix one verbose answer doesn't just shorten that answer — it shifts the model's behavior across the whole distribution. Now answers that genuinely needed detail get truncated. You fixed one case and silently regressed a class of others you never looked at.
Model changes are the same story from a different direction. A managed provider can update the model behind a stable alias. You didn't edit anything, but the output distribution moved, and behaviors you relied on may have drifted. Either way, the effect is non-local: a single change ripples across inputs you didn't test.
The consequence is that hand-checking a few examples is structurally unable to catch regressions. You'd have to manually re-verify the entire input space on every change, which is impossible. This is exactly the problem regression testing was invented for in normal software — and exactly why an LLM system needs its own version of it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Without eval-in-CI | With eval-in-CI |
|---|---|
| 'This prompt feels better' | Aggregate score on representative cases |
| Regressions found by users in prod | Regressions fail the build pre-release |
| Silent model updates go unnoticed | Scheduled run flags the drift |
| Fix one case, break unknown others | Per-case diff shows what regressed |
Real products, models, and research that use this idea.
- promptfoo running an eval suite as a CI step that fails the build when a prompt change drops the pass rate.
- LangSmith / Braintrust gating prompt and model changes behind dataset scores before merge.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep your held-out eval set from going stale as the product evolves?
Describe a loop that mines production failures and thumbs-down cases into new eval items, plus periodic review to retire obsolete cases.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Shipping a prompt tweak because it fixed the one case you were staring at, never checking the dozens of cases it silently broke.
60 second bullets to scan on the way to the call.
Why prompt changes have non-local effects across the input distribution
Why behavior drift is invisible without a measurement
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.