You have 500KB of relevant documentation that exceeds the 200K-token context window of your production model. How should you choose between context stuffing (fit what you can) and RAG retrieval?

The short answer is RAG. The reason is that 'fits in the window' and 'is the right thing to send' are not the same question, and 500KB of documentation is on the wrong side of that distinction. On cost: 500KB is roughly 125K tokens. Every single request you make to the model would pay for those 125K input tokens, even when the user asks a question that only needs three paragraphs to answer. Frontier-model pricing scales linearly with input tokens, so this is a real ongoing cost, not a one-time setup tax. On quality: long-context evals consistently show a recall degradation pattern called lost-in-the-middle. The model attends well to content near the start and end of the prompt, and less well to content stuck in the middle. Stuffing 125K tokens of docs means the most relevant paragraph for any given query has roughly a 50% chance of landing in the dead zone. On latency: input tokens drive prefill time. A 125K-token prefill is multiple seconds even on the fastest providers, versus a few hundred milliseconds for a 10K-token RAG context. RAG retrieves the 5 to 10K tokens that match the query, places them where the model attends best, costs roughly a tenth as much per call, and gives you citation-quality grounding for free. The narrow case where stuffing wins is when the corpus is small (under 50K tokens) and every query genuinely needs most of it; 500KB of docs is not that case.

The decision is driven by three quantities: cost per call, recall on the relevant content, and latency. **Cost.** 500KB is about 125K tokens. At current frontier pricing (Claude Opus 4.7 around $5 per million input tokens, GPT-5.5 around $1.25 to $2.50) stuffing costs roughly $0.20 to $0.65 per call before the response is generated, whether or not the question actually needs the whole corpus. RAG retrieving 5 to 10K tokens per query collapses that to under $0.15, a 10x to 25x reduction. **Recall.** Long-context evals (Anthropic's needle-in-a-haystack, Nelson Liu's lost-in-the-middle, Greg Kamradt's pressure tests) all show that content placed in the middle of a long context is recalled worse than content at the start or end. A 125K-token prompt has a large 'middle' and any given relevant paragraph has roughly even odds of landing there. RAG retrieves the relevant chunks and places them in the high-attention positions. **Latency.** Prefill scales linearly with input tokens. A 125K-token prefill is multiple seconds even on tuned vLLM stacks. A 10K-token RAG context prefills in hundreds of milliseconds. The operational answer is RAG with hybrid retrieval (BM25 plus dense embeddings, re-ranked by a cross-encoder), 5 to 10 chunks of 500 to 1000 tokens each, and citations rendered alongside the answer. The narrow case where stuffing wins is a small corpus (under 50K tokens) where every query needs most of the content and the corpus is stable enough that prefix caching can amortize the cost. 500KB of documentation is not that case.