Walk through the playbook when a model vendor announces a deprecation deadline on your pinned model

Short answer·Medium·4.0 · 0·~3 min·Asked atCharacter Ai Doordash Flowise

Attempt it

Anthropic emails that claude-3-opus-20240229 will be sunset in 90 days. You have it pinned in production for one of your flagship features. Walk through the playbook.

Free · 2 AI evals / day

A **model deprecation** is the vendor's announcement that a specific model snapshot will stop serving on a date. Anthropic, OpenAI, and Google all do this on a published schedule (usually 6-12 months for hosted models, 90 days for older snapshots). The instinct of a new engineer is to do a one-line PR that changes the model id and ship it. That fails for two reasons. First, the new model behaves differently: refusal rate, verbosity, tool-calling format, and tokenization all shift between snapshots. The prompt that worked on Opus 3 will not score the same on Opus 4.7 without tuning. Second, you usually do not know every place the model id appears: it might be hardcoded in three services, referenced in a prompt registry, and listed as a fallback in the LiteLLM gateway config. The playbook treats the window as a project, not a config change. Inventory the surface area, shadow-test candidates on real traffic, tune the prompt for the new model, run eval-CI to prove no regression, then canary roll with the old model held as the gateway fallback until the actual sunset date.

**Day 0, inventory.** Grep the monorepo for the model id, query the LLM gateway (LiteLLM, Portkey) logs for any call referencing it, and walk the prompt registry to find indirect references. Surface area is almost always larger than the team's mental model. **Week 1, shadow.** Pick two or three candidate replacements (Claude Opus 4.7, Sonnet 4.5, or a non-Anthropic option) and route a copy of production traffic to each in shadow mode. Score with **LLM-as-judge** on paired old vs new outputs, plus refusal-rate delta, format-validity, latency, and cost per request. The shadow run answers the only question that matters: does this replacement preserve quality on your actual workload. **Weeks 2-4, prompt-tune.** Major version bumps almost always need prompt edits. Verbosity caps, refusal phrasing, tool-call schema, and chain-of-thought behavior all shift. Iterate against the golden set in eval-CI; gate merges on no-regression across the rubric. **Weeks 5-6, canary.** Roll the new model to 5%, then 25%, then 100% with the standard soak between steps. Watch cohort regressions, not just aggregate quality: a 2% mean drop with a 30% drop in one tenant is a different fire. **Weeks 7-13, keep the fallback warm.** Leave the deprecated model pinned in the gateway as a fallback until the actual sunset timestamp. If something regresses in production, the fallback recovers instantly. **Postmortem.** Document what surprised you: gaps in inventory, golden-set coverage, prompt-tuning effort. Next deprecation is cheaper.

TL;DR

Inventory call sites, shadow candidates, prompt-tune, canary roll, keep the old model warm as fallback until the actual sunset date, then postmortem.

Memory aid

Easy to grasp

Imagine your landlord says the building closes in 90 days. You do not just grab the first apartment listing. First you walk every room and write down what is here. Then you tour a few candidate places and live a weekend in each one (shadow mode). You pick the best fit and slowly move your stuff over (canary). You keep the old keys until the actual closing day in case something breaks. After you settle, you write down what you wish you had done differently so the next move is easier. Model deprecations work the same way: inventory, shadow, tune, canary, fallback, postmortem. Surprise sunsets only break teams that try to rush all of those steps at once.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Model deprecation is the most predictable surprise in production LLM ops. Vendors retire snapshots on a schedule, and every team running pinned model ids will receive that email eventually. The teams that handle it without a fire are not the ones with the best engineers; they are the ones with a documented playbook the second the email arrives.

This walkthrough sequences the six gates of that playbook against a concrete scenario: Anthropic announces a 90-day sunset on a Claude Opus snapshot you depend on for a flagship feature. The same shape applies to OpenAI deprecating dated gpt-4o snapshots, Google retiring Gemini 1.x, or Bedrock dropping a specific Llama checkpoint.

Gate 1, Inventory the call sites

The first hour is reconnaissance, not engineering. The model id leaks into more surfaces than the team expects.

Grep the monorepo for the literal string. Then query the LLM gateway (LiteLLM, Portkey, Vercel AI Gateway) logs for the past 30 days and aggregate by model field; any caller you missed in code shows up here. Walk the prompt registry: many registries store a default model per prompt, and that resolution happens at runtime so it does not appear in grep. Check the eval harness configuration; a deprecated model still serving golden-set runs is a silent half-migration. Finally, check internal tooling and docs: the snippet in the onboarding wiki tells the next engineer to use the old id.

The deliverable from this gate is a checklist of every reference, with an owner per surface. Migration cannot begin until that list is complete, because the rollback story depends on knowing where to point traffic back if the new model misbehaves.

Gates 2 and 3, Shadow and prompt-tune

Gates 4 and 5, Canary ramp and the fallback discipline

Gate 6, Postmortem and runbook update

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic sunsetting older Claude 3 snapshots on a published schedule, pushing teams to migrate to Claude Opus 4.7 or Sonnet 4.5.
OpenAI deprecating earlier-generation snapshots in favor of GPT-5.5, requiring downstream services to ramp through canary on dated ids.

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through how you would build the shadow-mode harness so it scores quality without doubling your inference bill forever.

Sample a fixed percentage of production traffic (1-5% is usually enough for statistical power within a week), tee the request to both old and new models, log paired outputs to Langfuse or Braintrust, score with LLM-as-judge in a nightly batch (not inline), and shut the shadow off the moment canary starts. The harness lives behind a feature flag so it can be re-enabled on the next deprecation without rebuilding.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the swap as a single PR. The deprecation window exists precisely because prompt tuning, eval-CI sign-off, and canary observation each need real calendar time.

60 second bullets to scan on the way to the call.

Where the deprecated model id can hide (code, prompt registry, gateway fallback, eval harness)
Why shadow mode goes before canary and what signals to score

Primary sources. Browse if you want the original framing.