**The context window** is the maximum total number of tokens a model can handle in one API call. It is measured in tokens, not in characters or words.
The most important thing to understand is what counts against that budget. Everything counts. The system prompt counts. The user's current message counts. The full conversation history (every prior turn) counts. Any retrieved documents you injected for RAG count. Any tool-call results count. And critically, the model's own reply also counts.
So when Claude Opus 4.7 or Gemini 3.1 advertise a 1M-token context window, that does not mean you can send a 1M-token prompt. It means the input and the output combined must fit under 1M tokens. If your input is already 999,000 tokens, the model has only 1,000 tokens of room to reply.
The practical consequence: you have to budget context. Long conversations need history compression. RAG pipelines need to limit retrieved chunks. Long documents need chunking or summarization. The window is a hard ceiling.
**The context window** is a single token budget that the model has to fit an entire call into, including its own reply. It is the most consequential operational constraint of any LLM application.
What counts against the budget: the system prompt (often 500 to 2,000 tokens for a real product), the few-shot examples if you have them, the full conversation history (every user message and every assistant reply, growing linearly with turns), any retrieved context from a RAG pipeline (3 to 10 chunks of 500 tokens each is typical), any tool-call results returned to the model, and the generated reply itself.
The critical implication is that input and output share the budget. A 1M-token window on Claude Opus 4.7 or Gemini 3.1 means input plus output must fit under 1M, not that the input alone can be 1M. The max-tokens parameter caps the reply length, and the model will refuse to start generating if there is not enough room left.
Most client SDKs handle context-window exhaustion by truncating from the oldest messages, which silently drops earlier context. Production systems instrument token usage per call as a metric and apply explicit history summarization, RAG chunk limits, or conversation windowing before the SDK has to truncate. The newer 1M-token windows have not eliminated this discipline; they have just moved the ceiling higher.
Quick reference
TL;DR
The context window is the maximum total tokens the model can process in one call, covering system prompt, user message, history, retrieved context, and the generated reply as one shared budget.
Imagine a whiteboard with a fixed number of inches of writing space. Everything for one conversation has to fit on that whiteboard at once. The system instructions go at the top, then the conversation so far, then any reference notes you pasted in, and at the very end the model writes its reply. If the whiteboard is six feet wide, all of that combined has to fit in six feet. If your system prompt and history already use five feet eleven inches, the model only has one inch left to answer. Buying a wider whiteboard helps, but you still cannot fill the entire whiteboard with input and expect room for the model to write. The model's reply shares the same whiteboard as everything else.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking a model purely on context-window size when your actual prompts use only 5K tokens The difference between a 128K window and a 1M window is irrelevant if you never approach either; optimize for quality and cost per call instead.
Filling the window with as much RAG context as fits More context typically hurts faithfulness and always costs more; production RAG ships 3 to 10 chunks, not 100.
Treating long-context windows as a substitute for retrieval over a large corpus Loading a million-token corpus into every call is wildly cost-inefficient and slower than vector search; retrieval then generate beats stuff the window for most production use cases.
Quick scan · 2–4 min
Everything important, quickly.
6 min: define context window as total token budget, enumerate what counts against it, explain input plus output share the budget, describe truncation and lost-in-the-middle, name 2026 frontier window sizes.
Real-world examples
Real products, models, and research that use this idea.
1Claude Opus 4.7, Gemini 3.1, and GPT-5.5 all ship around 1M-token API windows; older open-weight Llama variants offer 8K to 128K.
2Anthropic's prompt-caching docs explicitly note that cached system prompts still count against the context window; caching reduces cost, not budget.
3Lost-in-the-middle benchmarks (Liu et al. 2023, follow-ups in 2024-2025) show frontier models recall facts in the middle of a long context less reliably than at the start or end.
4Cursor and Cody both run conversation-history compaction once a chat session crosses a watermark in the window, rather than relying on SDK truncation.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the lost-in-the-middle effect, and how does it interact with the advertised context-window size?
Sign in or go Pro to see the approach for this follow-up.
QHow does prompt caching change the cost economics of a long context window?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the difference between extending a context window via fine-tuning versus via positional-embedding tricks like RoPE scaling?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the context window as input-only and forgetting that the model's reply consumes the same budget, then getting truncated outputs when the system prompt and history already filled most of the window.