You're migrating a production prompt from Claude Opus 4.7 to GPT-5.5 to cut cost. What's the most reliable indicator that you can or cannot move the prompt as is?
Run the prompt unchanged on both models on your golden eval set and compare per-metric retention. 95%+ on faithfulness, format, and refusal correctness means it ports; a meaningful drop means over-tuning to one model.
Imagine you have a recipe that works perfectly in your oven and you are about to buy a new oven from a different brand. The only honest way to know whether the recipe transfers is to bake the same dish in both ovens and compare. Tasting one cookie from the new oven and saying it tastes fine is the failure mode. Comparing twenty cookies on flavor, texture, and color is how you actually know. Same with prompts and models: vibe checks lie, eval scores do not.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt portability across frontier models is one of those topics where the right answer feels boring and the wrong answer feels rigorous. The boring right answer is to run an eval. The wrong but fancy answers are to compare context windows, read provider guides, or trust published benchmarks. None of the fancy answers actually tell you whether your prompt will work on the new model.
The reason eval is the only real answer is that prompt-model coupling is empirical. Two frontier models trained by different labs on different data with different post-training pipelines will diverge on specific axes that are invisible until you measure them on your task. The published benchmarks measure tasks that are usually different from yours. The provider guides give general heuristics, which are necessary input but not evidence. The only ground truth is your prompt on your task on the new model with the metrics you care about.
The deep dive walks through what a portability eval suite contains, how per-metric retention is the right comparison shape, the documented axis by axis divergences between Claude and GPT models, the production rollout pattern that catches what the offline eval missed, and the cost vs quality tradeoff that determines whether a small retention drop is acceptable.
What a portability eval suite actually contains
The suite has three layers. First, the representative golden set: 100 to 300 queries sampled from real production traffic, each with a known-good output from the current model. This is the bulk of the eval and measures average-case retention.
Second, the adversarial slice: 20 to 50 known-tricky queries that the current prompt handles correctly. These are cases where the prompt only works because of specific model behavior, where a small phrasing edit would break it, or where the failure mode is silent. The adversarial slice is what catches portability regressions that the average case hides.
Third, the regression cases: outputs that the current production system got wrong but that have been fixed; these should stay fixed on the new model. Skipping this layer means a successful migration on metrics A and B can quietly reintroduce regressions on metric C.
The metrics are application-specific. Faithfulness against retrieved context for RAG. Format compliance against a schema for structured output. Refusal correctness on adversarial out of scope queries. End-task accuracy. Latency. Cost. Each metric gets bootstrap CIs so you can distinguish a real drop from sampling noise.
The whole suite lives in version control and runs in CI. Every model upgrade across providers triggers a full re-run, and the results are posted as a PR comment with a per-metric retention table. That is the artifact that gates the migration.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Cursor's engineering posts describe running the same coding prompts against Claude Opus 4.7 and GPT-5.5 on a golden code-edit set and routing by model strength per metric.
- Perplexity's stack runs side by side evals on Claude and GPT-5.5 prompts and tags every call with the model snapshot, enabling per-model regression attribution.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does a credible portability eval suite look like end to end?
Golden set of 100 to 300 representative queries plus a 20 to 50 query adversarial slice; 3 to 5 orthogonal metrics with bootstrap CIs; held-out test split; CI-runnable; results posted as a PR comment with per-metric retention table.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Eyeballing five outputs on the new model, declaring portability based on the vibe, and missing a 15 percent drop in format compliance that breaks downstream parsers on the cases you did not check.
60 second bullets to scan on the way to the call.
Golden-set composition for a portability eval (representative + adversarial)
Per-metric retention as the right comparison shape
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.