You route easy turns to a small model (Claude Haiku 4.5 or GPT-5 nano) and hard turns to a large model (Claude Opus 4.7 or GPT-5). How should your context budget differ between the two routes, and what concrete numbers would you start with?
Free · 2 AI evals / day
**The two routes are not symmetric.** A small model and a large model in 2026 may both advertise 200K token windows, but their effective long-context quality is very different. Long-context evals like RULER and BABILong consistently show small variants of every family losing 15-25 accuracy points by 32K tokens, while flagship variants stay near their short-context baseline out to 64K and beyond.
That asymmetry should drive an asymmetric budget. For the small-model route (Claude Haiku 4.5, GPT-5 nano, Gemini 3.1 Flash), keep retrieval tight: top-3 chunks after a rerank, roughly 4-8K tokens. Summarize conversation history aggressively to 2-4K tokens. Keep the system prompt stable to maximize prompt-cache hits.
For the large-model route (Claude Opus 4.7, GPT-5, Gemini 3.1 Pro), retrieval can stretch to 16-32K with top-10 or top-15 chunks. History can hold up to 8K with lighter summarization. The system prompt remains the same so caching benefits both routes.
The key discipline: do not assume the flagship's tolerance for more context is a license to use it. Even Opus benefits from a tighter context when the task is well-scoped. The per-route eval curve, not the marketing limit, is what should set each cap.
**Model-aware budgeting starts from one empirical fact: nominal capacity is not effective capacity, and the gap is wider for small models.** RULER and BABILong both show small variants degrading 15-25 points by 32K tokens while flagships hold near baseline to 64K. Any budget that ignores that gap will overserve the small route and bill the large route for tokens that do not help.
The slot by slot allocation I would start with:
- **Small route (Haiku-class, GPT-5 nano-class).** System: 1-2K, cached. Retrieval: 4-8K, top-3 reranked. History: 2-4K, aggressive summary. Tool results: bounded to last 2 with eviction. Total target: 8-15K.
- **Large route (Opus-class, GPT-5 flagship).** System: 1-2K, cached. Retrieval: 16-32K, top-10 to top-15. History: 8K, lighter summary. Tool results: last 3-5 with eviction. Total target: 30-50K.
Two invariants apply to both routes. **The system prompt is identical** so prompt caching amortizes across both routes. **The eval curve sets the cap**, I would run a per-route RULER-shape eval at 4K / 8K / 16K / 32K and pick the elbow where accuracy starts dropping more than two points per doubling.
One trap to avoid: Opus tolerating more context is not the same as Opus benefiting from it. The cheapest, fastest, highest-quality option is the one that puts the least into context that still answers the question, on either route.
Quick reference
TL;DR
Give the small-model route a tighter context budget than the large-model route because small models lose accuracy faster as context grows; size both budgets from per-route eval curves, not from advertised window limits.
Picture two chefs sharing the same kitchen. One is a quick line cook who handles simple orders fast; the other is a head chef who tackles the complicated dishes. If you dump twenty ingredients on each cutting board, the line cook gets overwhelmed and starts mixing the wrong ones. The head chef can still find the right ingredients but works slower and bills you for the extra effort. So you prep less for the line cook and more for the head chef. The kitchen and the recipes are the same; the prep is different by station. Small and large language models work the same way: the small one needs a tighter, cleaner counter to perform at its best.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-model deployments where there is no routing between size classes Asymmetric budgeting is a routing-time decision; without two routes there is one budget to set and the question collapses to a normal context-budget exercise.
Workloads where every turn requires the full corpus (legal review of a contract, multi-document summarization) If the task itself requires the long context, the budget cannot be shrunk on the small route without changing the task. Route everything to the large model and accept the cost.
Early-stage prototypes before per-route eval curves are measured Setting asymmetric numbers from intuition is worse than a conservative single budget. Run the eval first, then introduce the asymmetry once data justifies it.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the small vs large degradation gap from RULER, give slot by slot starting numbers for each route, defend the asymmetry, explain why per-route eval curves drive the cap, and note that tolerating context is not benefiting from it.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic publishes RULER results across Claude Opus 4.7, Sonnet 4.6, and Haiku 4.5 showing the small-model degradation gap that motivates asymmetric budgeting.
2OpenAI's GPT-5 nano versus GPT-5 flagship long-context evals in 2026 show the same family-internal pattern.
3Cohere Rerank 3 and Voyage Rerank-2 are the production rerankers most often paired with this two-route pattern in 2026.
4Routing libraries like RouteLLM and Portkey expose per-route context limits as first-class config, reflecting that asymmetric budgeting is now a standard production pattern.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle a mid-session route flip from Haiku to Opus when the context already exceeds the Haiku budget?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the per-route eval look like in practice and what is the elbow you are looking for?
Sign in or go Pro to see the approach for this follow-up.
QHow does this interact with prompt caching across routes?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Sizing both budgets from the advertised window limit. The hard cap is what the API accepts; the soft cap is where quality starts dropping, and the two diverge sharply for small models.