Flashcard: what role does the LLM's context window play in a RAG pipeline?

Attempt it

The context window is the total tokens the LLM can read in a single forward pass (e.g. 200K for Claude Opus 4.7, 1M for Gemini 3.1, 1M for GPT-5.5). In RAG, the prompt the LLM sees is: system instructions + retrieved chunks + the user query + room for the response. Every retrieved chunk consumes tokens, so the context window caps how many chunks you can include. In practice, even with million token windows, you do not fill the budget: latency and cost scale with input tokens, and 'lost-in-the-middle' attention degradation means more chunks does not always mean better answers. Typical production RAG passes 3-10 chunks, not 100, even when the window allows it.

Your answer

Quick reference

TL;DR

The context window is the hard cap on prompt tokens; in RAG it constrains how many chunks you can include, but cost, latency, and lost-in-the-middle keep production usage well under the cap.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of the context window as the size of the desk the model is reading at. Anything you want it to use has to fit on that desk: your instructions, the chunks you retrieved, the user's question, and space for the answer it will write. A bigger desk lets you spread out more pages, but it does not change the fact that the reader's eyes get tired in the middle of a crowded desk. Even when modern models have desks the size of a conference table, smart RAG systems still place only a handful of pages on the desk, the ones most likely to be useful. Stuffing the desk wastes time and money and actually makes the reader less reliable. The window sets the ceiling, but practice sets the working number, and the working number is much smaller.

Key concepts

RAG system design Chunking strategies

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define the context window, explain what occupies it in a RAG prompt, walk through the cost latency attention reasons production RAG passes few chunks, and call out lost-in-the-middle by name.

Real-world examples

Real products, models, and research that use this idea.

Claude Opus 4.7 shipping a 200K-token window, used in production RAG with typical prompt sizes well under 20K tokens per query.
Gemini 3.1 Pro offering a 1M-token window, used for full document context in code review and long-document QA workloads.
GPT-5.5 supporting 1M tokens of input; production RAG stacks still pass 5 to 10 chunks per query for cost and faithfulness.
Liu et al. Lost-in-the-Middle paper quantifying U-shaped recall on long contexts, still cited in 2026 RAG retrieval design.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through the budget math for a RAG prompt to a 1M-token Gemini call.

Sign in or go Pro to see the approach for this follow-up.

QDoes a 1M-token window make RAG obsolete? Why or why not?

Sign in or go Pro to see the approach for this follow-up.

QHow does lost-in-the-middle affect chunk ordering in a long RAG prompt?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Filling the context window with as many chunks as fit. More tokens cost more money and more latency, and the lost-in-the-middle effect means accuracy can actually drop past a small number of well chosen chunks.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.