What is the context window of an LLM?
The context window is the hard ceiling on prompt plus completion tokens the model can attend to in one call, fixed at training by the positional encoding scheme.
Imagine a desk where you can lay out a finite number of index cards at once. Every card is a token: words from the question, the conversation so far, plus the answer being written. The desk has a strict edge. Once the cards run past that edge, nothing more fits. The model cannot peek beyond the desk because it was only ever taught to read positions inside it. If you want to feed it a longer document, you either trim cards off the back, group several cards onto one space, or stretch the desk using a special technique designed for that. The desk size is decided when the model is built, not when you ask it a question, and it bundles both what you say and what it replies into the same finite space.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The context window is the most misunderstood number on an LLM spec sheet. People treat it as an input budget, but it covers everything the model sees plus everything it produces in a single call. A 128k context with a 30k-token prompt leaves 98k tokens for the reply, not another 128k.
More importantly, the cap is not a configuration choice the serving stack made. It is a structural property of the trained model. The transformer learns positions through a positional encoding scheme, and that scheme is only fit to the range it saw during training. Send a position the model has never seen and the attention layer has no information about where that token lives in the sequence.
This deep dive walks through what the window actually is, why it is a hard cap, how memory and quality degrade as you approach it, the three families of techniques that stretch it, and the product question that always comes back at you: when is a bigger window the wrong solution to your problem?
The goal by the end is to be able to discuss context window the way a serving engineer does. You should know what determines the ceiling, what it costs you in GPU memory and latency, what fails first as you push the limit, and when retrieval beats expansion. Those conversations come up in every system design loop touching long-context features.
What the window covers, in token units
The cap is total tokens, prompt plus completion, in one forward pass. If the model advertises 128k, that is the sum, not each side. A 100k-token prompt leaves at most 28k tokens of reply before the model is forced to stop.
The unit is the model's own token, not English words or characters. Tokenization varies by model family, but a rough rule for English is 1 token to 0.75 words. Code and non-English text tokenize more aggressively, often 1 token to 0.3 to 0.5 words, so a 10k-line Python file can blow past a 32k window faster than people expect.
The API typically rejects requests that exceed the cap up front, but more subtle failures appear long before you hit the limit. Recall in the middle of a long context degrades, output quality drops, and prefill latency grows linearly with input length. Treating the window as a clean usable budget all the way to the top is one of the most common production mistakes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Gemini 3.1 Pro advertises a 2 million token context window, used heavily for whole-codebase and long-video tasks.
- Claude Opus 4.7 ships a 1 million token window, with output capped well below the input cap to manage decode cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does long-context recall often degrade in the middle of the window even within the trained range?
Trace the lost-in-the-middle finding to attention head specialisation: recency and primacy heads dominate, so tokens in the middle of a long context get less attended on average. Counter it with retrieval over stuffing, and with training data that explicitly tests middle-position recall.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the context window as input-only. The cap covers prompt plus completion combined, so a long prompt eats the budget the model has left to actually answer.
60 second bullets to scan on the way to the call.
The definition of context window as prompt plus completion token cap
Why positional encoding makes the cap a hard training-time decision
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.