Order the stages of a production LLM quality assurance pipeline from development to release

Attempt it

1Online metric monitoring in full production
2Staging canary eval on a representative traffic slice
3Pre-merge CI quality gate blocking regression
4Prompt or model change is made by an engineer
5Offline golden-set eval against a frozen test set
6A/B test or shadow deployment measuring real-traffic quality

Quick reference

TL;DR

A change climbs an exposure ladder: engineer edit, offline eval, CI gate, staging canary, A/B test, then continuous full production monitoring.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine test-driving a new bus route before letting passengers ride. First the planner sketches the route on paper, which is the engineer's change. Then they drive it empty against a checklist of known stops, which is the offline eval. Next a supervisor signs off before it can go live, which is the CI gate. Then they run one quiet trip with a few volunteers, which is the canary. Then half the buses use the new route while half keep the old one, so you can compare results, which is the A/B test. Finally, once it works, every bus uses it and you watch the daily ridership dashboards forever, which is production monitoring. Each step costs more and exposes more real riders, so you never jump to crowded rush hour before the empty test drive.

Key concepts

Eval feedback loop Regression CI gate Canary eval Drift detection

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: walk the six stages in order, anchor each to its user-exposure level, and explain why the offline eval and CI gate must precede any live stage.

Real-world examples

Real products, models, and research that use this idea.

LangSmith and Braintrust both wire offline golden-set eval into CI as a pre-merge gate that blocks on score regression.
Promptfoo runs a frozen eval suite locally and in CI before any prompt change is allowed to merge.
Anthropic and OpenAI gate frontier model releases on internal eval suites, then staged canary, then phased traffic rollout.
Statsig and LaunchDarkly drive the A/B and shadow-deployment stage for LLM features with traffic splitting and guardrail metrics.
Langfuse and Arize monitor production proxy signals like thumbs and follow-up rate plus rolling canary eval to flag drift.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you keep the offline golden set from going stale as production traffic shifts?

Sign in or go Pro to see the approach for this follow-up.

QWhy is shadow deployment often safer than an A/B test, and when do you still need the A/B?

Sign in or go Pro to see the approach for this follow-up.

QWhat statistical test belongs in the CI gate, and why not just compare mean scores?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Running the offline eval after deploy, or skipping the staging canary, so a regression reaches real users before any cheap automated gate had a chance to catch it.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.