Design an eval framework that tells you whether a production prompt survives a model migration (Claude 4.7 to 5.0, or Claude to GPT-5.5). What do you measure and what's the gate?
You're the MLOps engineer responsible for upgrading the LLM behind a production assistant. You need to migrate from Claude Opus 4.7 to either Claude Opus 5.0 (same family, new generation) or GPT-5.5 (different vendor). Design the evaluation framework that tells you whether the production prompt will survive the migration as-is. Cover: what goes into the golden set, what measurement axes you score, how you handle adversarial cases, what cost and latency you track, what the gate criterion looks like, and what the rollout pattern is if the eval passes.
Free · 2 AI evals / day
A portability eval framework is the test bench that tells you whether your production prompt will survive a model migration. The shape has five parts: the golden set, the measurement axes, the gate criterion, the adaptation playbook, and the rollout pattern.
**Golden set.** Three composition buckets. Representative cases (60% of the set, sampled from real traffic with known-good outputs from the current model). Adversarial cases (25%, known-tricky queries the current prompt handles correctly). Regression-fix cases (15%, things that were broken and have been fixed). Total size 150 to 300 queries.
**Measurement axes.** Five orthogonal axes. Faithfulness (does the output match retrieved context). Format compliance (does it follow the schema). Refusal correctness (does it refuse the right things). End-task accuracy (does it solve the user's request). Cost and latency. Each quality metric gets bootstrap 95% confidence intervals.
**Gate criterion.** Per-metric retention >= 95% on the tracked axes, with the lower CI bound above 90%. Zero new false refusals on the adversarial slice. All regression-fix cases still pass. Cost and latency at or below current. A miss on any axis means adaptation, not migration.
**Adaptation playbook.** If format compliance drops on GPT-5.5, swap XML tags for JSON. If instruction-following regresses, restructure system prompts as numbered rules. If refusal calibration shifts, tighten the refusal language. The eval localizes the regression so the adaptation is targeted.
**Rollout pattern.** Flag-rolled, 5 to 10% of traffic to the new model first, every call tagged with the model snapshot, auto-rollback rules pre-registered. Ramp slowly over a week if the live metrics confirm the offline retention.
Same-family upgrades (Claude 4.7 to 5.0) need a lighter eval than cross-vendor migrations (Claude to GPT-5.5) because the divergence axes are narrower within a family.
A defensible portability eval has three pieces: a layered golden set, per-metric retention with CIs, and a flag-rolled production rollout.
**Three-bucket golden set.** Representative cases (60%, 100 to 200 queries from real traffic) measure average-case retention. Adversarial slice (25%, 30 to 80 queries of known-tricky cases) catches portability regressions the average case hides. Regression-fix slice (15%, 20 to 50 cases) prevents reintroducing fixed bugs. Versioned in git, quarterly refresh, 30% held-out test split.
**Five orthogonal axes.** Faithfulness (LLM-as-judge + citation check). Format compliance (JSON schema validation). Refusal correctness (in-scope and out of scope probes). End-task accuracy (task rubric). Cost and latency separately. Each quality metric gets a retention ratio (new model / current model on the same cases) with bootstrap 95% CIs.
**Gate criterion.** Per-metric retention >= 95% on each axis with CI lower bound above 90%. Zero new false refusals on the adversarial slice. All regression-fix cases pass. Cost at or below current; latency p50 and p95 at or below (or within 20% if the cost saving justifies). Aggregate win-rate is not the gate; per-axis retention is.
**Axis-specific adaptation.** Format compliance regression on GPT-5.5 usually means swap XML tags for JSON or markdown headers. Instruction-following regression means restructure multi-paragraph prompts as numbered rules. Refusal regression means tighten or loosen refusal language plus few-shot demos. The eval localizes; adaptation is targeted.
**Production rollout.** Flag-rolled, model-attributed traces. 5-10% of traffic to the new model keyed off a stable hash. Pre-registered auto-rollback rules: any tracked metric below 95% retention for two consecutive hours flips back. Ramp 5% -> 25% -> 50% -> 100% over a week.
**Same-family vs cross-vendor.** Same-family upgrades (Claude 4.7 to 5.0) need a lighter eval. Cross-vendor (Claude to GPT-5.5) needs the full heavyweight.
Quick reference
TL;DR
Three-bucket golden set (representative, adversarial, regression-fix), five axes with per-metric retention + CIs, gate at 95%+ plus zero new false refusals, flag-rolled rollout with auto-rollback.
Imagine you are about to switch suppliers for a part that goes into your product. The right way to decide is not to try one part and call it good. You build a test bench with three kinds of tests: typical cases, the cases that broke last time, and the cases that are designed to be sneaky. You measure five things on each test: does the part fit, does it last, does it pass safety, does it cost less, does it work fast. You only switch if the new part passes every test within tight tolerances. Then you switch slowly on a small fraction of products first and watch the real-world data. That is what a portability eval framework is for prompts.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The migration is for an exploratory prototype with no production users Heavyweight eval infrastructure is overhead; spot-check on representative queries and iterate by hand until the prompt shape stabilizes.
You do not yet have a golden eval set for the current model You cannot do a credible portability check without one; build the golden set against the current model first, then port the framework to the new model.
The migration is forced by an end of life deprecation with no acceptable alternative Eval still matters for adaptation guidance, but the gate criterion shifts from 'migrate only if it ports' to 'adapt until it ports'; the question becomes about finding the prompt that works, not about whether to switch.
Same-family minor revision (snapshot bump within a release) A small smoke test on representative queries usually suffices; the full eval is overkill for changes that the provider's release notes describe as non-behavioral.
Quick scan · 2–4 min
Everything important, quickly.
6 min: three-bucket golden set + five orthogonal axes + per-metric retention with CIs + pre-registered gate criterion + axis-specific adaptation playbook + flag-rolled rollout + same-family vs cross-vendor distinction.
Real-world examples
Real products, models, and research that use this idea.
1Cursor's engineering posts describe running the same coding prompts against Claude Opus 4.7 and GPT-5.5 on a golden code-edit set, reporting per-metric retention, and routing each request to the model that wins on the specific axis the request needs.
2Perplexity's stack runs a side by side eval suite for Claude and GPT-5.5 on its production search prompts, tags every call with the model snapshot, and rolls migrations behind feature flags with auto-rollback on metric regression.
3Anthropic's eval cookbook and OpenAI's Evals framework both ship reference implementations of golden set based portability evals with bootstrap CIs and CI integration; Braintrust and Langfuse layer prompt versioning on top.
4Notion AI's documented migration from Claude Sonnet 4.x to a multi-model router including Gemini 3.1 Flash used a three-bucket golden set and per-metric retention to decide which traffic slices were safe to route to each model.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build the adversarial slice when you do not yet know which cases will be tricky on the new model?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right LLM-as-judge configuration for cross-model evals to avoid judge bias?
Sign in or go Pro to see the approach for this follow-up.
QWhen the eval surfaces a 10-point drop on format compliance, what is the iteration cycle?
Sign in or go Pro to see the approach for this follow-up.
QHow do you decide between a single-model migration and a multi-model router?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes for a same-family upgrade like Claude 4.7 to 5.0?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Trusting a passing aggregate eval on the representative slice while the adversarial slice quietly regresses, then shipping a migration that breaks the long tail of edge cases that the average user never hits but the high-value users do.