You have 500KB of relevant documentation that exceeds the 200K-token context window of your production model. How should you choose between context stuffing (fit what you can) and RAG retrieval?

MCQ·Medium·4.0 · 0·~1 min·Relevant atAnthropic OpenAI

Attempt it

The short answer is RAG. The reason is that 'fits in the window' and 'is the right thing to send' are not the same question, and 500KB of documentation is on the wrong side of that distinction. On cost: 500KB is roughly 125K tokens. Every single request you make to the model would pay for those 125K input tokens, even when the user asks a question that only needs three paragraphs to answer. Frontier-model pricing scales linearly with input tokens, so this is a real ongoing cost, not a one-time setup tax. On quality: long-context evals consistently show a recall degradation pattern called lost-in-the-middle. The model attends well to content near the start and end of the prompt, and less well to content stuck in the middle. Stuffing 125K tokens of docs means the most relevant paragraph for any given query has roughly a 50% chance of landing in the dead zone. On latency: input tokens drive prefill time. A 125K-token prefill is multiple seconds even on the fastest providers, versus a few hundred milliseconds for a 10K-token RAG context. RAG retrieves the 5 to 10K tokens that match the query, places them where the model attends best, costs roughly a tenth as much per call, and gives you citation-quality grounding for free. The narrow case where stuffing wins is when the corpus is small (under 50K tokens) and every query genuinely needs most of it; 500KB of docs is not that case.

The decision is driven by three quantities: cost per call, recall on the relevant content, and latency. **Cost.** 500KB is about 125K tokens. At current frontier pricing (Claude Opus 4.7 around $5 per million input tokens, GPT-5.5 around $1.25 to $2.50) stuffing costs roughly $0.20 to $0.65 per call before the response is generated, whether or not the question actually needs the whole corpus. RAG retrieving 5 to 10K tokens per query collapses that to under $0.15, a 10x to 25x reduction. **Recall.** Long-context evals (Anthropic's needle-in-a-haystack, Nelson Liu's lost-in-the-middle, Greg Kamradt's pressure tests) all show that content placed in the middle of a long context is recalled worse than content at the start or end. A 125K-token prompt has a large 'middle' and any given relevant paragraph has roughly even odds of landing there. RAG retrieves the relevant chunks and places them in the high-attention positions. **Latency.** Prefill scales linearly with input tokens. A 125K-token prefill is multiple seconds even on tuned vLLM stacks. A 10K-token RAG context prefills in hundreds of milliseconds. The operational answer is RAG with hybrid retrieval (BM25 plus dense embeddings, re-ranked by a cross-encoder), 5 to 10 chunks of 500 to 1000 tokens each, and citations rendered alongside the answer. The narrow case where stuffing wins is a small corpus (under 50K tokens) where every query needs most of the content and the corpus is stable enough that prefix caching can amortize the cost. 500KB of documentation is not that case.

Quick reference

TL;DR

500KB ~= 125K tokens. It fits a 200K window but pays full cost per call and hits lost-in-the-middle. RAG retrieves only 5K-10K relevant tokens per query, cutting cost 10x to 25x and keeping high-recall positions.

Memory aid

Easy to grasp

Imagine you have a 1,200-page reference book and somebody asks you a question. You could carry the whole book to the meeting every time, but it is heavy, you only read a few pages, and you tend to forget what was on the pages in the middle. RAG is the librarian who hands you only the three pages relevant to the question. Context stuffing is dragging the whole book. The book strategy works if it is a thin pamphlet you read every time, but for a real reference, the librarian wins on weight, on cost, and on actually remembering what you read.

Key concepts

Retrieval-Augmented Generation Lost-in-the-middle Context window LLM cost optimization

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: token math on 500KB + cost per call + lost-in-the-middle + prefill latency + when stuffing actually wins + production RAG pattern.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's prompt-engineering docs explicitly recommend RAG over stuffing for Claude Opus 4.7 once the corpus exceeds a small fraction of the 200K window, citing both cost and recall on the middle of the context.
Perplexity routes every search through dense plus BM25 retrieval before invoking Claude or GPT-5.5, because stuffing the web index is structurally impossible and even a daily slice would dilute attention.
Cursor's repo-aware features retrieve a handful of relevant files at request time rather than stuffing the entire codebase into Claude Opus 4.7, which is how the per-edit latency stays interactive.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen does prompt caching change the math in favor of stuffing?

QWhat does a credible RAG eval look like for this kind of system?

QHow do you handle the case where the relevant content spans many chunks?

QHow does lost-in-the-middle interact with prompt-position strategy?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating a 200K window as license to stuff the whole corpus, paying for 125K tokens per call, and shipping a system that quietly misses facts buried in the middle of the prompt.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.