Contrast the hard token budget with the soft quality budget of a context window

Every long-context model has two distinct budgets that engineers need to track separately. The **hard budget** is the API's enforced token limit. For a model like Claude Sonnet 4.5 it is 200K input tokens; for Gemini 3.1 Pro it is 2M; for GPT-5 it is around 400K depending on the SKU. Exceed it and the API returns an error. The hard limit is binary: either you fit or you do not. The **soft budget** is the size beyond which adding more tokens reduces the quality of the answer, even though the API still accepts the call. The model becomes less accurate, slower to reason, and more likely to miss instructions buried in the middle of a long prompt. The soft budget is set by accuracy, not by the API. The soft budget is usually much smaller than the hard limit. For retrieval-heavy question answering on a 200K-window model, the soft budget is often 8K to 32K. Past that, accuracy drops measurably even though the call still succeeds. Context engineering is the discipline of staying inside the soft budget. The hard budget exists as a safety net for the rare cases where you genuinely need every token, not as a target to fill up to.

The **hard budget** is the API-enforced token limit and is binary. Claude Sonnet 4.5 accepts 200K input tokens; Gemini 3.1 Pro accepts 2M; GPT-5 accepts around 400K. Exceed the hard budget and the API rejects the call. The **soft budget** is the token size beyond which marginal added context degrades answer quality, even though the API still accepts the call. The soft budget is task-dependent, model-dependent, and usually much smaller than the hard limit. For retrieval QA, it tends to sit at 8K to 32K on a 200K-window model. For agentic trajectories, it can be even smaller, in the 4K to 16K range, because attention has to spread across plan, tool history, and current step. The two budgets fail differently. Hitting the hard limit produces a clean error you can detect and route around. Crossing the soft budget produces silent quality loss: the model still answers, but the answer is more likely to miss instructions, hallucinate details from low-attention regions, or pick the wrong document out of a retrieval set. The practical implication is that the hard budget is a safety net, not a target. A team that designs against the 200K hard limit and stuffs prompts to 100K will ship a system that is slower, costlier, and less accurate than one that designs against an 8K-32K soft limit. Long-context evals such as Needle-in-a-Haystack, RULER, and BABILong measure exactly the curve that defines the soft budget, and they are the right tool for sizing it for a given model and task.