You're the MLOps engineer responsible for upgrading the LLM behind a production assistant. You need to migrate from Claude Opus 4.7 to either Claude Opus 5.0 (same family, new generation) or GPT-5.5 (different provider). Design the evaluation framework that tells you whether the production prompt will survive the migration as is. Cover: what goes into the golden set, what measurement axes you score, how you handle adversarial cases, what cost and latency you track, what the gate criterion looks like, and what the rollout pattern is if the eval passes.
Three-bucket golden set (representative, adversarial, regression-fix), five axes with per-metric retention + CIs, gate at 95%+ plus zero new false refusals, flag-rolled rollout with auto-rollback.
Imagine you are about to switch suppliers for a part that goes into your product. The right way to decide is not to try one part and call it good. You build a test bench with three kinds of tests: typical cases, the cases that broke last time, and the cases that are designed to be sneaky. You measure five things on each test: does the part fit, does it last, does it pass safety, does it cost less, does it work fast. You only switch if the new part passes every test within tight tolerances. Then you switch slowly on a small fraction of products first and watch the real-world data. That is what a portability eval framework is for prompts.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Portability evals are where most LLM teams either build a defensible migration practice or accumulate years of fear-driven indecision about model upgrades. The teams that ship migrations confidently have an eval framework that tells them, with numbers and confidence intervals, whether the new model retains the behavior they need. The teams that do not have such a framework either avoid migrations or migrate on intuition. The first pays a quality and cost penalty as the model market evolves around them. The second pays a quality penalty when the silent regressions hit production.
The framework has five key pieces: a layered golden set, orthogonal measurement axes, a pre-registered gate criterion, an axis-specific adaptation playbook, and a flag-rolled production rollout with auto-rollback.
The three-bucket golden set
Representative cases (60%, 100 to 200 queries) are sampled from real production traffic with known-good outputs from the current model. They measure average-case retention: 'does the new model do the typical job as well as the current model?'. Without this bucket, the eval is not anchored to the actual production distribution.
The adversarial slice (25%, 30 to 80 queries) is the most important bucket for portability. It contains known-tricky cases the current prompt handles correctly: edge phrasings, refusal-correctness probes, format-compliance stress tests, cases that depend on specific model behaviors like Claude's instruction-following on multi-paragraph system prompts. The adversarial slice is what catches the portability regressions that the average case hides. A migration can pass on representative cases and still fail on the adversarial slice. Those failures hurt the long-tail user experience, the safety surface, or the format-strict downstream parsers.
The regression-fix slice (15%, 20 to 50 queries) contains cases that were broken in production and have been fixed. These must continue to pass on the new model. Without this bucket, a migration can quietly reintroduce previously fixed bugs. The team only discovers it weeks later, when the same user reports the same problem twice.
The whole set lives in version control alongside the prompt, with a held-out test split (30%) that is never used during iteration. Quarterly refresh from production traffic keeps the set anchored to the current distribution. Treating the eval set as versioned code is what makes the framework work at scale.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Promptfoo runs the same prompts and test cases against several providers in one config, tracks metrics for each provider separately, and has assertion types for JSON schema validity, LLM rubric grading, cost and latency, which covers most of the per-axis comparison.
- Feature-flag platforms such as LaunchDarkly can send a percentage of traffic to the new model keyed on a stable user ID and roll it back with one flag change, which is the rollout half of the framework.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build the adversarial slice when you do not yet know which cases will be tricky on the new model?
Seed it with known production failures of the current model, with edge phrasings discovered during iteration, with refusal probes from your safety team, and with format stress tests; grow it from live failures of the new model once the rollout starts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Trusting a passing aggregate eval on the representative slice while the adversarial slice quietly regresses. The shipped migration then breaks the long tail of edge cases that the average user never hits but the high-value users do.
60 second bullets to scan on the way to the call.
Three-bucket golden-set composition with concrete percentages and sizes
Five orthogonal measurement axes and how each is scored
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.