The hard budget is the API-enforced token limit (200K input, 8K output for a typical Claude or Gemini SKU). Exceed it and the call fails. The soft budget is the size beyond which marginal added tokens reduce answer quality even though the API still accepts them. The soft budget is task-dependent, model-dependent, and usually much smaller than the hard limit, frequently 8-32K for retrieval-heavy QA on a 200K-window model. Context engineering is largely the practice of staying inside the soft budget while the hard budget remains as a safety net.
Every long-context model has two distinct budgets that engineers need to track separately.
The **hard budget** is the API's enforced token limit. For a model like Claude Sonnet 4.5 it is 200K input tokens; for Gemini 3.1 Pro it is 2M; for GPT-5 it is around 400K depending on the SKU. Exceed it and the API returns an error. The hard limit is binary: either you fit or you do not.
The **soft budget** is the size beyond which adding more tokens reduces the quality of the answer, even though the API still accepts the call. The model becomes less accurate, slower to reason, and more likely to miss instructions buried in the middle of a long prompt. The soft budget is set by accuracy, not by the API.
The soft budget is usually much smaller than the hard limit. For retrieval-heavy question answering on a 200K-window model, the soft budget is often 8K to 32K. Past that, accuracy drops measurably even though the call still succeeds.
Context engineering is the discipline of staying inside the soft budget. The hard budget exists as a safety net for the rare cases where you genuinely need every token, not as a target to fill up to.
The **hard budget** is the API-enforced token limit and is binary. Claude Sonnet 4.5 accepts 200K input tokens; Gemini 3.1 Pro accepts 2M; GPT-5 accepts around 400K. Exceed the hard budget and the API rejects the call.
The **soft budget** is the token size beyond which marginal added context degrades answer quality, even though the API still accepts the call. The soft budget is task-dependent, model-dependent, and usually much smaller than the hard limit. For retrieval QA, it tends to sit at 8K to 32K on a 200K-window model. For agentic trajectories, it can be even smaller, in the 4K to 16K range, because attention has to spread across plan, tool history, and current step.
The two budgets fail differently. Hitting the hard limit produces a clean error you can detect and route around. Crossing the soft budget produces silent quality loss: the model still answers, but the answer is more likely to miss instructions, hallucinate details from low-attention regions, or pick the wrong document out of a retrieval set.
The practical implication is that the hard budget is a safety net, not a target. A team that designs against the 200K hard limit and stuffs prompts to 100K will ship a system that is slower, costlier, and less accurate than one that designs against an 8K-32K soft limit. Long-context evals such as Needle-in-a-Haystack, RULER, and BABILong measure exactly the curve that defines the soft budget, and they are the right tool for sizing it for a given model and task.
Quick reference
TL;DR
The hard budget is the API token limit and is binary. The soft budget is the size beyond which quality degrades even though the API still accepts the call, and it is usually far smaller.
Imagine a backpack with a zipper. The zipper closes at a fixed maximum: that is the hard limit, and if you try to stuff more in, the zipper will not close. But long before you reach that physical maximum, the bag becomes too heavy to carry comfortably. Your shoulders start to ache, you walk slower, you trip on stairs. That comfort limit is the soft budget. The model has both. The zipper is enforced by the API. The shoulder pain is the model's accuracy quietly dropping as you cram more context in. Smart packing means staying inside the comfortable limit, not just inside the zipper limit.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Indivisible documents where chunking would destroy cross-reference signal (a single legal contract, an entire codebase for cross-file review). Here you must use the hard budget because the soft-budget approach (retrieve and trim) would lose the global structure the task requires. Accept the quality tradeoff and plan extra evaluation around it.
Tasks where measurement infrastructure is impossible (one-off internal tools with no users). If you cannot measure the soft budget, applying a strict soft-budget regime is guesswork. Use conservative defaults (8K-16K) and rely on hard-budget tolerance for resilience.
Streaming or interactive UX where any rejection is unacceptable. Strictly enforcing a small soft budget can cause the system to drop content the user expected to be included. In these cases the hard budget gives you headroom while you negotiate UX-level tradeoffs with the product team.
Quick scan · 2–4 min
Everything important, quickly.
Open with the binary vs gradual distinction. Give a concrete 2026 model example for each budget. Explain why the soft budget is task and model dependent. Name the evaluation tools that measure it. Close with the principle that context engineering is soft-budget engineering, with the hard budget as a safety net.
Real-world examples
Real products, models, and research that use this idea.
1Claude Opus 4.7 advertises 200K input but Anthropic's own long-context guide recommends 32K-64K for best quality on retrieval tasks
2Gemini 3.1 Pro offers a 2M+ window but Google's RULER results show quality erosion past 128K
3OpenAI's GPT-5.5 supports up to 1M (tier-dependent) but the Responses API docs advise summary and tool patterns rather than dumping documents
4The Needle-in-a-Haystack benchmark from Greg Kamradt visualized the soft budget curve for the first time in 2023 and is still the reference eval
5Voyage v3 embeddings + Cohere rerank-3.5 are tuned to deliver good top-5 results under 8K context, the soft-budget sweet spot for most RAG
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you empirically measure the soft budget for a given model and task?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the soft budget shrink for agent trajectories compared to single-shot QA?
Sign in or go Pro to see the approach for this follow-up.
QWhen does it make sense to deliberately operate near the hard budget?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the hard token limit as the only budget that matters and stuffing context up to that ceiling, accepting silent quality loss.