The context window is the hard cap on prompt tokens; in RAG it constrains how many chunks you can include, but cost, latency, and lost-in-the-middle keep production usage well under the cap.
Think of the context window as the size of the desk the model is reading at. Anything you want it to use has to fit on that desk: your instructions, the chunks you retrieved, the user's question, and space for the answer it will write. A bigger desk lets you spread out more pages, but it does not change the fact that the reader's eyes get tired in the middle of a crowded desk. Even when modern models have desks the size of a conference table, smart RAG systems still place only a handful of pages on the desk, the ones most likely to be useful. Stuffing the desk wastes time and money and actually makes the reader less reliable. The window sets the ceiling, but practice sets the working number, and the working number is much smaller.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The context window question separates candidates who treat RAG as a slogan from candidates who have run it in production. The slogan answer says 'the context window limits how many chunks you can retrieve' and stops. The production answer says 'the window is the ceiling, but cost, latency, and attention quality are the constraints that actually shape the design, and they push you to ship far fewer chunks than the window allows.'
This matters because the 2026 frontier models broke the old constraint. When GPT-3 had 2K tokens and GPT-3.5 had 4K, the window genuinely was the design driver: every byte counted, chunks had to be small, and prompts were carefully whittled. With Claude Opus 4.7 at 200K and Gemini 3.1 Pro and GPT-5.5 at 1M, that constraint relaxed dramatically. Candidates who anchor on the old picture say things like 'context windows are huge now so RAG is going away.' That is the wrong conclusion, and explaining why it is wrong is what the question is testing for.
This deep dive defines the context window precisely, walks through what occupies it in a real RAG prompt, explains the three stacked constraints (window, cost, attention) and which one binds first, and lands on the 2026 production pattern of over-retrieve then rerank then ship few.
What the context window is, precisely
The context window is the maximum number of tokens an LLM can read and reason over in a single forward pass. It is a property of the model architecture, set by how the attention mechanism scales and how the model was trained for long-context behavior. In 2026, the headline numbers are Claude Opus 4.7 at 200K tokens, Gemini 3.1 Pro and Pro Vision at 1M tokens, GPT-5.5 at 1M tokens, and most open weight Llama 4 variants at 128K tokens.
The window covers everything in the prompt plus space for the model's output. You do not get the full 1M for input alone; the response also comes out of that budget. A 1M-token model serving a 100K-token output therefore has 900K of input headroom, not the full million.
For RAG specifically, the prompt contains four things in the typical structure: the system prompt (instructions, persona, grounding directives), the retrieved chunks (the evidence the model reads), the conversation history (in multi-turn settings, the prior turns), and the user query (the current question). Add the reserved output budget and the four pieces have to total at most the window size. Most RAG systems are nowhere near this ceiling. A production prompt at 5 chunks of 600 tokens plus 2K of system instructions plus 200 tokens of query plus 2K for the response totals about 7K tokens. That is 3 percent of a 200K window and 0.7 percent of a 1M window.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Claude Opus 4.7 shipping a 200K-token window, used in production RAG with typical prompt sizes well under 20K tokens per query.
- Gemini 3.1 Pro offering a 1M-token window, used for full document context in code review and long-document QA workloads.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through the budget math for a RAG prompt to a 1M-token Gemini call.
System prompt plus conversation history plus k chunks of m tokens plus query plus response budget; show how the response budget shrinks if input grows.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Filling the context window with as many chunks as fit. More tokens cost more money and more latency, and the lost-in-the-middle effect means accuracy can actually drop past a small number of well chosen chunks.
60 second bullets to scan on the way to the call.
What the context window is and how 2026 model limits compare
What occupies the window besides retrieved chunks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.