Order the stages of a production LLM quality assurance pipeline from development to release

A production **LLM quality pipeline** is a staircase: each step costs more and exposes more real users than the one before it. You want to catch problems on the cheap, low-exposure steps before they reach paying customers. It starts when an engineer makes a change to a prompt or swaps in a new model. That change is just code at this point, with zero user exposure. The first automated gate is an offline eval against a frozen golden set: a fixed list of test queries with known-good expectations. It runs on the developer's branch and costs only API calls. Next the change hits a pre-merge CI gate that blocks the merge if scores regress versus the baseline. After merge, a staging canary runs the new version on a small representative slice of traffic to catch problems the static eval missed. Then an A/B test or shadow deployment compares the new version against the old on real traffic at partial exposure. Only after that succeeds does the change reach full production, where online metrics like thumbs and follow-up rate get monitored continuously to catch drift.

The ordering principle is **escalating user exposure**. Each stage costs more and touches more real users than the last, so you gate cheaply and early. The sequence runs: 1. An engineer makes the prompt or model change. It is just a diff, zero exposure. 2. Offline golden-set eval runs against a frozen test set on the branch. 3. A pre-merge CI quality gate blocks the merge if scores regress. 4. A staging canary evaluates a small representative traffic slice. 5. An A/B test or shadow deployment compares old versus new on real traffic. 6. Online metric monitoring watches the change continuously in full production. Why this order? Offline eval is fastest and has zero exposure, so it goes first. The CI gate is the cheapest blocking check before code lands. The canary catches what a static set cannot: live distribution shifts. The A/B test gives a controlled comparison at partial exposure. Monitoring is the only ongoing stage, because it never ends. The inversion to avoid is shipping first and evaluating later. That lets a regression reach users before any gate fires. You always pay the cheap checks before the expensive, high-exposure ones.