You're the MLOps engineer responsible for upgrading the LLM behind a production assistant. You need to migrate from Claude Opus 4.7 to either Claude Opus 5.0 (same family, new generation) or GPT-5.5 (different vendor). Design the evaluation framework that tells you whether the production prompt will survive the migration as is. Cover: what goes into the golden set, what measurement axes you score, how you handle adversarial cases, what cost and latency you track, what the gate criterion looks like, and what the rollout pattern is if the eval passes.
Three-bucket golden set (representative, adversarial, regression-fix), five axes with per-metric retention + CIs, gate at 95%+ plus zero new false refusals, flag-rolled rollout with auto-rollback.
Imagine you are about to switch suppliers for a part that goes into your product. The right way to decide is not to try one part and call it good. You build a test bench with three kinds of tests: typical cases, the cases that broke last time, and the cases that are designed to be sneaky. You measure five things on each test: does the part fit, does it last, does it pass safety, does it cost less, does it work fast. You only switch if the new part passes every test within tight tolerances. Then you switch slowly on a small fraction of products first and watch the real-world data. That is what a portability eval framework is for prompts.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Portability evals are where most LLM teams either build a defensible migration practice or accumulate years of fear-driven indecision about model upgrades. The teams that ship migrations confidently have an eval framework that tells them, with numbers and confidence intervals, whether the new model retains the behavior they need. The teams that do not have such a framework either avoid migrations (and pay a quality and cost penalty as the model market evolves around them) or migrate on vibes (and pay a quality penalty when the silent regressions hit production).
The framework has five load-bearing pieces: a layered golden set, orthogonal measurement axes, a pre-registered gate criterion, an axis-specific adaptation playbook, and a flag-rolled production rollout with auto-rollback. Each piece has a specific job and they compose into a complete answer.
The deep dive walks through each piece, then through the cross-vendor divergence patterns that the framework is designed to surface, and finally through the operational distinction between same-family upgrades (lighter eval) and cross-vendor migrations (heavier eval) that matters at scale.
The three-bucket golden set
Representative cases (60%, 100 to 200 queries) are sampled from real production traffic with known-good outputs from the current model. They measure average-case retention: 'does the new model do the typical job as well as the current model?'. Without this bucket, the eval is not anchored to the actual production distribution.
The adversarial slice (25%, 30 to 80 queries) is the most important bucket for portability. It contains known-tricky cases the current prompt handles correctly: edge phrasings, refusal-correctness probes, format-compliance stress tests, cases that depend on specific model behaviors like Claude's instruction-following on multi-paragraph system prompts. The adversarial slice is what catches the portability regressions that the average case hides. A migration can pass on representative cases and fail on the adversarial slice in ways that hurt the long-tail user experience, the safety surface, or the format-strict downstream parsers.
The regression-fix slice (15%, 20 to 50 queries) contains cases that were broken in production and have been fixed. These must continue to pass on the new model. Without this bucket, a migration can quietly reintroduce previously-fixed bugs and the team only discovers it weeks later when the same user reports the same problem twice.
The whole set lives in version control alongside the prompt, with a held-out test split (30%) that is never used during iteration. Quarterly refresh from production traffic keeps the set anchored to the current distribution. The eval set as versioned code discipline is what makes the framework operational at scale.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Cursor's engineering posts describe running the same coding prompts against Claude Opus 4.7 and GPT-5.5 on a golden code-edit set, reporting per-metric retention, and routing each request to the model that wins on the specific axis the request needs.
- Perplexity's stack runs a side by side eval suite for Claude and GPT-5.5 on its production search prompts, tags every call with the model snapshot, and rolls migrations behind feature flags with auto-rollback on metric regression.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build the adversarial slice when you do not yet know which cases will be tricky on the new model?
Seed it with known production failures of the current model, with edge phrasings discovered during iteration, with refusal probes from your safety team, and with format stress tests; grow it from live failures of the new model once the rollout starts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Trusting a passing aggregate eval on the representative slice while the adversarial slice quietly regresses, then shipping a migration that breaks the long tail of edge cases that the average user never hits but the high-value users do.
60 second bullets to scan on the way to the call.
Three-bucket golden-set composition with concrete percentages and sizes
Five orthogonal measurement axes and how each is scored
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.