Why is context stuffing the opposite of context engineering, even on a 1M-token window?

Flashcard·Easy·4.0 · 0·~30s·Asked atMckinsey Moveworks Mu Sigma

Attempt it

Context engineering is about putting the right tokens in the right slots. Dumping the whole KB does the opposite: it puts every token everywhere. Cost scales linearly with input tokens. Latency scales with input length. The lost-in-the-middle curve still applies. The right chunk is now buried in the middle of nearly a million tokens of distractors. The model spreads attention thinly, accuracy drops, and you have paid full price for the privilege. A large window is permission to be less aggressive about retrieval, not permission to skip it.

Your answer

Context engineering is the discipline of choosing what reaches the model's context window and how it is structured. Context is a scarce resource even when the window is large. Dumping the entire knowledge base into context goes against this in three concrete ways. **Cost.** API providers charge per input token. A 1M-token prompt costs roughly 125 times more than an 8K-token prompt on the same model. If you run that on every query, your cost per query becomes too high. **Latency.** Time to first token scales with input length. A 1M-token prompt takes seconds before the model even starts generating. Users notice. **Accuracy.** The lost-in-the-middle effect is a real, measured phenomenon. Long-context models use the start and end of the input best and often miss content in the middle. Burying the right chunk in the middle of 999,000 other tokens makes it harder for the model to use, not easier. A large window gives you a higher ceiling for legitimate cases (long documents, long conversations). It is not permission to skip retrieval.

**Context engineering** treats the context window as a scarce resource and decides what reaches it. Stuffing inverts every choice that discipline makes. The first issue is cost. Input tokens are billed at the input rate, and that rate does not drop for long prompts. A query that should have been 8K input tokens but runs at 1M is paying a 125x penalty on every call. Prompt caching only amortizes the static portion; the dynamic chunks pay full price every turn. The second issue is latency. Time to first token grows at least linearly with input length, and faster at very long lengths because attention cost is quadratic. A 1M-token prefill is multiple seconds before the model emits a single output token. For interactive applications that is fatal. The third and most counter-intuitive issue is accuracy. The lost-in-the-middle effect is well documented by Liu et al. It shows that models use the head and tail of long contexts best. The middle is under-attended. Long-context benchmarks such as RULER and NoLiMa also show accuracy falling well before the advertised window. Stuffing buries the right chunk in the middle of nearly a million tokens of distractors and gives the model permission to spread attention thinly. Accuracy on retrieval and reasoning tasks drops measurably. A 1M-token window is a higher hard budget. The soft budget, the point past which more tokens stop helping, is usually a small fraction of that. Good context engineering targets that soft budget.

TL;DR

A 1M-token window raises the hard ceiling, not the soft one. Stuffing pays full cost and full latency to bury the right chunk under distractors.

Memory aid

Easy to grasp

Picture a library researcher with a question. One librarian walks them straight to the three books that answer it. Another librarian wheels in every book in the library and dumps the entire pile on the table. The second approach is not more thorough. It is just more expensive and more confusing. The researcher's eyes have to scan more, and the right page is harder to find. The answer comes out blurrier because so many irrelevant pages were in view. A bigger reading table does not change this. It just makes the pile fit. Context engineering is the first librarian. Stuffing is the second.

Key concepts

Context engineering Retrieval-augmented generation Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Frontier context windows of 1M tokens or more make it tempting to argue that retrieval is a relic of the 8K-window era. The argument is a category error. A larger window expands the hard budget (the API limit). The soft budget (the size past which added tokens stop helping or start hurting) is usually a small fraction of that.

The two budgets

A modern context window has two budgets, not one.

The hard budget is the token ceiling the API will accept. On current Claude models that is 1M tokens. Past the ceiling the request fails outright. Most production prompts do not get anywhere near this.

The soft budget is the size past which marginal tokens reduce answer quality. This is task-dependent and model-dependent. Long-context evals like RULER and BABILong consistently show that effective accuracy declines well before the hard ceiling, often by 100K tokens or earlier on many tasks.

Why the gap exists

Attention is a finite resource even when the window is generous. The model has to decide where to look. A million tokens of input gives it many places to look other than the right one. The lost-in-the-middle curve is the empirical signature of this. The model attends well to the head and tail of a long input and underattends to the middle. The right chunk in the middle of a million-token stuffed prompt is easy for the model to miss.

Cost: the linear scaling

Latency: prefill dominates

Accuracy: the lost-in-the-middle curve

When large windows are genuinely useful

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Claude Opus 4.7 ships a 1M-token context window, and Anthropic's guidance still recommends retrieval over stuffing for large production knowledge bases.
Gemini models offer long context windows, but long-context benchmarks such as RULER and BABILong show effective accuracy declining well before the limit.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen a 1M-token window actually helps, what does the right usage look like?

Long-document QA where the document is the input, long agent trajectories with genuine state, conversations whose history would otherwise need summarization. The win is fewer compromises, not more retrieval.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating advertised window size as effective capacity. Frontier models have nominal 200K to 2M windows. But their soft budget, where added tokens stop helping or start hurting, is usually a small fraction of that.

60 second bullets to scan on the way to the call.

Define context engineering in one sentence
List the three axes on which stuffing fails (cost, latency, accuracy)

Primary sources. Browse if you want the original framing.