Adding more retrieved evidence to a prompt has effects beyond just 'more context'. Trace the impact on cost, latency, and quality, and explain why all three usually move against you past a certain point.
Free · 2 AI evals / day
When you add more retrieved evidence to a prompt, three things move against you, not just one.
The first is **cost**. Providers bill per input token. If you send 32k tokens of context instead of 8k, you pay roughly four times as much for the input portion of the call, before the model has generated a single output token. Prompt caching can soften this for the static parts of the prompt that do not change between turns, but the retrieved chunks are different every turn so caching does not help there.
The second is **latency**. Before any model can produce its first output token, it must run a prefill phase over every input token. That phase scales roughly linearly with input length. A 5x larger prompt typically produces a 3-5x longer time to first token. For chat interfaces, this is felt directly as a slower response.
The third is **quality**. Past a soft budget that is usually well below the hard window limit, accuracy starts dropping. This is the lost-in-the-middle effect plus distractor dilution: mediocre chunks weaken the salience of the strong ones, and the model has more places to get distracted.
The critical observation is that past the soft budget all three move against you at once. More tokens, longer waits, worse answers. That is why the right rule is to stop adding chunks at the point where the next chunk is unlikely to help, not at the point where the window literally fills up.
Every extra token of context creates a **three-way penalty** on cost, latency, and quality, and all three curves bend against you past a soft threshold that sits well below the hard window limit.
**Cost** is the simplest curve. Providers bill input tokens at a published rate. Sending 32k tokens of context costs four times what 8k does on the input side, before the output is generated. Prompt caching mitigates this for the **byte-stable portion** of the prompt, system instructions, tool definitions, persistent memory, but not for the retrieved chunks, which change every turn. So the variable portion that grows with retrieval pays full price every call.
**Latency** is dominated by the prefill phase. Before the first output token streams, the model must encode every input token. Prefill is compute-bound and scales roughly linearly with input length on most 2026 providers (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro). A 5x larger prompt typically increases time to first token by 3-5x. For agents this multiplies across many iterations; for chat it is felt directly as a slower response.
**Quality** is the curve that surprises people. Past a soft budget, accuracy declines because of two compounding effects. **Lost-in-the-middle**: salient content placed in the middle of a long context is recalled less reliably than the same content at the ends. **Distractor dilution**: adding mediocre chunks does not just add noise, it actively weakens the model's attention on the strong chunks. The model has more places to be distracted, and the right answer becomes one signal among many.
The property that makes this hard is that past the soft budget all three curves move the wrong way together. You pay more, you wait longer, and you get a worse answer. The right policy is not 'fill the window' but 'stop at the marginal-utility point', usually a much smaller context.
Quick reference
TL;DR
Every extra token costs more dollars at the input rate, adds prefill latency to time to first token, and dilutes signal so accuracy drops past the soft budget, three curves bending against you in unison.
Imagine ordering a pizza with toppings. The first few toppings make it better. The next few make it heavier (more expensive), slower to bake (longer to deliver), and harder to taste any single flavor. Past some point, every new topping makes the pizza worse in all three ways at once. Adding context to a prompt works the same way. Some context helps the model give a great answer. But each extra chunk costs more money, takes more time to process, and starts hiding the good chunks behind mediocre ones. The trick is to stop adding toppings before any of those three things turns against you.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
When the evidence really is small and fits comfortably below the soft budget If the retrieval set is small enough that quality has not plateaued, you are on the rising part of the curve. Add chunks freely until the plateau.
Single-pass batch jobs where latency does not matter Batch inference removes the TTFT axis from the tradeoff. The cost and quality curves still apply, but the user-felt latency does not.
Synthetic-data generation where quality is judged externally If quality is measured by downstream consumers and not by the immediate LLM output, the in-context quality penalty matters less and you can push closer to the hard limit.
Quick scan · 2–4 min
Everything important, quickly.
Walk through the three curves in order: cost (input token billing, partial mitigation via caching), latency (prefill phase scaling with input length), quality (lost-in-the-middle plus distractor dilution). Make the key point that past the soft budget all three move against you simultaneously. Recommend evaluation across context sizes to find the plateau and budget there rather than at the hard window limit.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt caching docs show roughly 90 percent input cost reduction on cache hits, but only for byte-stable prefixes
2OpenAI GPT-5.5's prefill latency scales near-linearly past 16k input tokens in published benchmarks
3Pinecone and Weaviate's 2026 RAG guides recommend k=4-6 chunks as the typical quality plateau
4LangChain's RAG evaluation cookbook explicitly compares quality across context sizes to find the plateau
5Gemini 3.1 Pro's 2M-token window does not eliminate lost-in-the-middle, the dip is shallower but still measurable
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the tradeoff change with KV cache reuse across turns of a single conversation?
Sign in or go Pro to see the approach for this follow-up.
QWhy is the quality curve U-shaped rather than monotonically increasing or decreasing?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right operational metric to monitor for this tradeoff in production?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Maxing out the context window with whatever the retriever returns, on the assumption that more evidence is always better, and being surprised when both costs and error rates climb together.