Order the steps of a typical deploy a prompt change lifecycle from edit to full rollout

Order steps·Medium·4.0 · 0·~1 min·Asked atSalesforce Tcs Unity

Attempt it

1Rollout ramps to 25, 50, 100 percent on green metrics, or auto rolls back on a regression
2Reviewer approves; merge to main publishes a new prompt version id
3Online dashboard watches judge scores, refusal rate, and cost per request for a soak window
4Canary router sends 5 percent of production traffic to the new version
5Eval CI replays the golden set on both old and new prompts; bot posts the metric delta as a PR comment
6Developer edits the prompt template on a feature branch and opens a PR

A prompt change is a production behavior change, so it goes through the same kind of pipeline as a code deploy, with a safety gate built in. The standard six-step order: 1. **Developer edits the prompt** on a feature branch and opens a PR. The diff is visible in code review. 2. **Eval-CI runs the golden set** on both the old and new prompts and posts the metric delta as a PR comment. Reviewers see exactly what changed in measurable terms. 3. **Reviewer approves, merge to main** publishes a new prompt version id. The id is immutable and goes on every future trace. 4. **Canary router sends 5 percent of traffic** to the new version. The other 95 percent still hit the old one. 5. **Online dashboard watches** judge scores, refusal rate, and cost per request for a soak window (typically 30 minutes to a few hours). 6. **Rollout ramps to 25, 50, 100 percent** on green metrics, or auto-rolls back if any metric breaches its threshold. The two steps that often get confused are merge and canary. Merge creates the version id but does not route real traffic; the canary router is what flips production behavior.

**The lifecycle mirrors a code-deploy pipeline with a measurement gate in the middle.** Six steps in order: **1. Edit and PR.** Developer modifies the prompt template on a feature branch, opens a PR. The diff is reviewable as text. **2. Eval-CI.** A GitHub Action (or equivalent) replays the golden set on both prompts, runs an LLM-as-judge or rule-based scorer, computes deltas, runs a significance test, and posts the comparison as a PR comment. Merge is blocked if any tracked metric drops past threshold. **3. Merge to main.** The version id is minted at merge. If you content-hash, it is `sha256(prompt_text)`; if you registry-version, it is `summarize@v17`. The id is immutable and becomes the trace identifier. **4. Canary at 5 percent.** A feature flag (Statsig, LaunchDarkly, the registry's own router) routes a deterministic 5 percent of traffic to the new version with sticky user-bucketing so a single user does not flip mid-session. **5. Soak window with dashboards.** Online evals on sampled production traces compute judge scores; the dashboard also tracks refusal rate, cost per request, latency p99, and error rate. The soak length depends on traffic volume (high-volume consumer: 30 to 90 minutes; lower-volume B2B: hours to a day). **6. Ramp or rollback.** If all metrics hold, ramp to 25, then 50, then 100 percent on cooling-off windows between each step. If any metric breaches, the canary controller flips the flag back to the old version automatically and pages the on-call. The common confusion is between step 3 and step 4. Merge mints the id but no real user has seen it. The canary router is what actually exposes the new version to traffic.

TL;DR

The prompt-deploy loop mirrors a code deploy with a safety gate in the middle: PR, eval-CI, merge (which mints the version id), canary, dashboard soak, then ramp or auto-rollback.

Memory aid

Easy to grasp

Think of how a chef rolls out a new menu item. First they write down the new recipe on paper (the PR). Then a taster tries it next to the old version and writes a comparison note (eval-CI). The head chef reads the note and approves; the kitchen prints the new recipe card and stamps it with a number (the version id). The restaurant serves the new dish to a few tables (canary). The maitre d watches for empty plates and complaints (the dashboard). If those tables are happy, the new dish goes to more tables, then to everyone. If anyone sends it back, the kitchen instantly switches back to the old recipe. Every step is small, reversible, and visible.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The deploy-a prompt change lifecycle is the central loop of LLMOps. It looks superficially like a code deploy, and it borrows much of that machinery, but the artifact moving through the pipeline is a prompt diff and the safety gates are statistical rather than exact.

This walkthrough goes through the six standard steps in order, explains what each gate is actually protecting against, and surfaces the two places teams most often get confused: where the version id is created, and what the canary router does that merge does not.

Mental model: the loop has three gates (offline eval, online soak, commit ramp) bracketing one one-way door (merge mints the immutable id). Each gate rejects a different class of regression.

Steps 1 and 2: edit, PR, eval-CI

The change-management entry point

A prompt change starts as a text edit on a feature branch. The developer opens a PR; reviewers can read the diff as plain text. This is the only step that involves human judgment on the substance of the change.

For teams using a prompt registry, the edit is to a YAML or JSON file in the repo that the registry imports. For teams using a registry UI (LangSmith Hub, PromptLayer), the diff still lives in version control via export, and the PR is on the exported file.

Eval-CI as the offline gate

The key step that distinguishes LLMOps from DevOps: a CI job replays the golden set on both the old and new prompts. Standard implementation:

GitHub Actions (or equivalent) triggers on PR open and on every subsequent push.
The job runs the eval framework (Promptfoo, DeepEval, Ragas, Inspect AI) with both prompts as candidates.
An LLM-as-judge or rule-based scorer produces per-example scores.
Aggregation reduces to a small set of tracked metrics: groundedness, helpfulness, refusal rate, format-compliance.
A significance test (bootstrap CI, paired t-test) compares old versus new.
A bot posts the comparison as a PR comment with deltas and confidence intervals.
Merge is blocked if any tracked metric drops more than its threshold.

Why eval-CI catches a specific failure class

The golden set is curated to cover known-important cases: representative happy paths, known regressions, edge cases that previously broke. A change that regresses any of these gets caught before merge. What the gate does not catch is regressions on the long tail of production traffic not represented in the golden set; those are the job of the next gate.

Step 3: merge mints the version id

Steps 4 and 5: canary and soak

Step 6: ramp or auto-rollback

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GitHub Actions plus Promptfoo or DeepEval is the canonical eval-CI stack used by teams shipping on Anthropic and OpenAI APIs in 2026.
LangSmith and Langfuse both expose canary routing primitives; Vercel AI Gateway adds a percentage-based router that flips without redeploy.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design the canary router to support per-tenant rollouts rather than global percentages?

Move from a percentage flag to a targeting rule: hash on tenant id, support allow-lists and deny-lists per tenant. Statsig and LaunchDarkly support this natively. The audit record gains a tenant-scope field.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating merge and rollout as the same step. The version id is created at merge, but real traffic only sees it after the canary router flips. Conflating the two hides the soak window where regressions surface.

60 second bullets to scan on the way to the call.

The six lifecycle steps in correct order
What happens at each gate (eval-CI, soak, commit)

Primary sources. Browse if you want the original framing.