Prompt caching reuses prefill KV for a byte-identical prefix, so cached input bills at roughly 10 percent of normal; any variable content above stable content breaks the hit.
Imagine reading the same long instruction sheet aloud before every conversation. The first time you read it slowly, but you remember it, so next time you can skip straight to the new question. Prompt caching is that memory for the model. When the start of your prompt is exactly the same as before, the provider reuses the work it already did and charges you a small fraction for those repeated tokens. But the trick only works from the very beginning. If you change even one word near the top, like stamping today's date, the model has to reread everything below it from scratch and you pay full price again. So you keep the unchanging part at the front and put the part that changes at the very end.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching is one of the highest-leverage cost levers in production LLM serving, and it is almost entirely a prompt-design decision rather than an infrastructure one. The core idea is simple: prefill, the phase where the model reads the whole prompt and computes a key and value for every token, is deterministic given the tokens. If the same prefix is sent again, the provider can reuse that prefill KV state instead of recomputing it.
Because the expensive compute already happened, vendors bill those reused input tokens at a steep discount. In 2026 the read rate sits around 10 percent of the normal input rate across Anthropic, OpenAI, and Google, with a small one-time write premium the first time a prefix is stored. You also get a latency win: skipping prefill on a long stable prefix slashes time to first token.
The entire benefit, however, hinges on one mechanic that trips up most candidates: the cache matches a byte-identical prefix from the very first token, and the hit ends at the first byte that differs. That single rule is what turns prompt caching from a checkbox into an architecture question. This deep dive covers why the discount exists, exactly how matching works, how to lay out a prompt to maximize the cached region, and the failure modes the multiple-choice distractors are built from.
Why prefill reuse earns a discount
A short-answer request has two cost phases. Prefill reads the entire prompt and computes keys and values for every token, filling the KV cache. Decode then generates output tokens one at a time. For a long prompt and a short answer, prefill dominates the input bill, because it touches every input token before a single output token appears.
Providers charge input tokens to cover that prefill compute. When the same prefix arrives again, the KV state is already computed, so the provider serves it from a cache tier rather than rerunning the matmuls. The skipped compute is what funds the discount. Crucially, prefill is deterministic: the same tokens always produce the same keys and values, so the cached state is a safe substitute for recomputing it, with no quality difference in the output.
Note that the discount is not 100 percent. The cached state still must be stored, fetched from a memory tier, and streamed into attention. That residual memory traffic is why the read rate lands near 10 percent rather than zero. You are paying for the bytes you move, not the math you skipped. This is also why caching is fundamentally a memory and bandwidth play, the same lens through which you reason about the underlying KV cache during ordinary decode.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude Opus 4.7 prompt caching bills cache reads at roughly 10 percent of input rate, with a one-time write premium, controlled by explicit cache breakpoints.
- OpenAI's GPT-5.5 applies automatic prefix caching for repeated prompt prefixes above a token threshold, discounting the cached input segment.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does cached input cost roughly 10 percent rather than zero?
The KV state still has to be stored, fetched from memory, and streamed into attention; it is not free. The discount reflects skipped prefill compute, not skipped memory traffic, so a residual read cost remains.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Putting a per-request value like a timestamp, session ID, or freshly retrieved chunks above the stable system prompt. It poisons the prefix and forces full-price prefill on everything below.
60 second bullets to scan on the way to the call.
Why cached input bills at roughly 10 percent of normal input rate
What the cache write premium is and when you pay it
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.