The context window is the maximum total tokens the model can process in one call, covering system prompt, user message, history, retrieved context, and the generated reply as one shared budget.
Imagine a whiteboard with a fixed number of inches of writing space. Everything for one conversation has to fit on that whiteboard at once. The system instructions go at the top, then the conversation so far, then any reference notes you pasted in, and at the very end the model writes its reply. If the whiteboard is six feet wide, all of that combined has to fit in six feet. If your system prompt and history already use five feet eleven inches, the model only has one inch left to answer. Buying a wider whiteboard helps, but you still cannot fill the entire whiteboard with input and expect room for the model to write. The model's reply shares the same whiteboard as everything else.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The context window is the single hardest budget that every LLM application has to plan around. Every other constraint (latency, cost, quality) is negotiable through engineering. The context window is a hard ceiling enforced by the model's architecture. Understanding what fills it and how it gets enforced separates engineers who have shipped LLM products from those who have only read about them.
This deep dive defines the context window precisely, walks through everything that counts against the budget in a real production call, explains how clients handle exhaustion, and connects the advertised window size to the practical effective context that determines retrieval quality.
What the context window actually is
The context window is the maximum total number of tokens (input plus output) the model can process in one forward pass. It is set by the model's architecture, specifically the trained range of its positional embeddings and the attention layer's sequence-length support. Once a call exceeds the window, the model literally cannot attend over the full sequence; it has to be truncated or extrapolated.
As of 2026, frontier model windows have grown dramatically. Claude Opus 4.7 ships at 1M tokens. Gemini 3.1 ships at 1M (with research versions pushing toward 10M). GPT-5.5 also ships at 1M via the API. Open-weight Llama models range from 8K (early variants) to 128K (Llama 3 long-context). The architectural levers that made this growth possible (RoPE positional embeddings, FlashAttention, ring attention) are why 2024-era models maxed out around 200K while 2026 models routinely ship at a million.
The advertised window is the architectural ceiling, but it is not the same as the practical effective context. We will get to that distinction in a later section. For now, the operational definition is the simple one: window size is the maximum total tokens (input + output) in one call.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Claude Opus 4.7, Gemini 3.1, and GPT-5.5 all ship around 1M-token API windows; older open-weight Llama variants offer 8K to 128K.
- Anthropic's prompt-caching docs explicitly note that cached system prompts still count against the context window; caching reduces cost, not budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the lost-in-the-middle effect, and how does it interact with the advertised context-window size?
Models recall facts in the middle of a long context unreliably compared to start or end. The advertised window is the architectural ceiling; the practical effective context for high-recall retrieval is smaller.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the context window as input-only and forgetting that the model's reply consumes the same budget, then getting truncated outputs when the system prompt and history already filled most of the window.
60 second bullets to scan on the way to the call.
What the context window is (max total tokens per call)
Everything that counts against it (system + user + history + RAG + tools + reply)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.