Design an eval framework that tells you whether a production prompt survives a model migration (Claude 4.7 to 5.0, or Claude to GPT-5.5). What do you measure and what's the gate?

A portability eval framework is the test bench that tells you whether your production prompt will survive a model migration. The shape has five parts: the golden set, the measurement axes, the gate criterion, the adaptation playbook, and the rollout pattern. **Golden set.** Three composition buckets. Representative cases (60% of the set, sampled from real traffic with known-good outputs from the current model). Adversarial cases (25%, known-tricky queries the current prompt handles correctly). Regression-fix cases (15%, things that were broken and have been fixed). Total size 150 to 300 queries. **Measurement axes.** Five orthogonal axes. Faithfulness (does the output match retrieved context). Format compliance (does it follow the schema). Refusal correctness (does it refuse the right things). End-task accuracy (does it solve the user's request). Cost and latency. Each quality metric gets bootstrap 95% confidence intervals. **Gate criterion.** Per-metric retention >= 95% on the tracked axes, with the lower CI bound above 90%. Zero new false refusals on the adversarial slice. All regression-fix cases still pass. Cost and latency at or below current. A miss on any axis means adaptation, not migration. **Adaptation playbook.** If format compliance drops on GPT-5.5, swap XML tags for JSON. If instruction-following regresses, restructure system prompts as numbered rules. If refusal calibration shifts, tighten the refusal language. The eval localizes the regression so the adaptation is targeted. **Rollout pattern.** Flag-rolled, 5 to 10% of traffic to the new model first, every call tagged with the model snapshot, auto-rollback rules pre-registered. Ramp slowly over a week if the live metrics confirm the offline retention. Same-family upgrades (Claude 4.7 to 5.0) need a lighter eval than cross-vendor migrations (Claude to GPT-5.5) because the divergence axes are narrower within a family.

A defensible portability eval has three pieces: a layered golden set, per-metric retention with CIs, and a flag-rolled production rollout. **Three-bucket golden set.** Representative cases (60%, 100 to 200 queries from real traffic) measure average-case retention. Adversarial slice (25%, 30 to 80 queries of known-tricky cases) catches portability regressions the average case hides. Regression-fix slice (15%, 20 to 50 cases) prevents reintroducing fixed bugs. Versioned in git, quarterly refresh, 30% held-out test split. **Five orthogonal axes.** Faithfulness (LLM-as-judge + citation check). Format compliance (JSON schema validation). Refusal correctness (in-scope and out of scope probes). End-task accuracy (task rubric). Cost and latency separately. Each quality metric gets a retention ratio (new model / current model on the same cases) with bootstrap 95% CIs. **Gate criterion.** Per-metric retention >= 95% on each axis with CI lower bound above 90%. Zero new false refusals on the adversarial slice. All regression-fix cases pass. Cost at or below current; latency p50 and p95 at or below (or within 20% if the cost saving justifies). Aggregate win-rate is not the gate; per-axis retention is. **Axis-specific adaptation.** Format compliance regression on GPT-5.5 usually means swap XML tags for JSON or markdown headers. Instruction-following regression means restructure multi-paragraph prompts as numbered rules. Refusal regression means tighten or loosen refusal language plus few-shot demos. The eval localizes; adaptation is targeted. **Production rollout.** Flag-rolled, model-attributed traces. 5-10% of traffic to the new model keyed off a stable hash. Pre-registered auto-rollback rules: any tracked metric below 95% retention for two consecutive hours flips back. Ramp 5% -> 25% -> 50% -> 100% over a week. **Same-family vs cross-vendor.** Same-family upgrades (Claude 4.7 to 5.0) need a lighter eval. Cross-vendor (Claude to GPT-5.5) needs the full heavyweight.