What does prefix caching store, and which workloads benefit most?
Explain what provider prompt (prefix) caching actually caches and the workload shape that makes it pay off.
Prefix caching stores the KV state of a stable leading prompt segment so repeat calls skip prefill over it; it pays off when many requests share a large, identical head.
Imagine a teacher who reads the same long set of class rules aloud before every lesson. Instead of re-reading the whole thing each time, she records it once and just plays the recording, then adds the day's new material live. Prompt caching does that for an LLM. The long, unchanging part at the start of your prompt — the standing instructions, a shared document — gets processed once and saved. Later requests that begin with the exact same opening skip re-processing it and pick up from where the saved part ends. You save time and money on the repeated part, but only if it stays word for word the same and sits at the very front.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching is one of the highest-leverage cost optimizations in LLM serving, and it's also one of the most misunderstood. The word "cache" pulls everyone's mental model toward HTTP: store the response, return it next time. That model is wrong here and leads to wrong design decisions, like expecting deterministic answers or putting the cache in the wrong layer.
What's actually cached is the intermediate computation, not the output. To see why that's the right thing to cache, you have to remember that an LLM call has two phases. Prefill reads the entire prompt and builds internal state before any output appears; decode then writes the answer one token at a time. Prefill over a long prefix is expensive and — crucially — produces the same internal state every time the prefix is identical. That repetition is exactly what a cache exists to exploit.
This walkthrough explains what the KV state is and why caching it is sound, spells out the exact-prefix requirement that turns prompt design into a layout problem, maps the workloads where it pays off, and closes with the economics and the caveats that decide whether it's a net win.
What actually gets cached: the KV state
Start with the mechanism. When the model runs prefill over a prompt, each layer projects every token into key and value vectors and stores them — collectively, the KV cache. Attention in every later step reads from these tensors. For a prompt of length P, building this state costs a full forward pass over all P tokens, which is the bulk of the time before the first output token appears.
Here's the key observation: that KV state is a pure function of the prompt tokens. If two requests start with the exact same sequence of tokens, the KV tensors for that leading segment are bit for bit identical. There's no randomness in prefill — sampling only enters during decode. So the leading prefix's state is perfectly reusable across requests.
Prefix caching memoizes precisely that. The provider runs prefill once, retains the KV tensors for the stable leading segment, and on the next matching request reuses them instead of recomputing. The model then runs prefill only over the new suffix — the part that differs — and proceeds to decode as normal.
The consequence to internalize: this caches a computation, not an answer. Every request still decodes fresh output, so responses remain non-deterministic and reflect the full prompt including the changing suffix. Anyone who describes it as "caching the response for a repeated question" has the wrong layer entirely.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Prefix caching | HTTP-style response cache |
|---|---|---|
| What's stored | KV state of the prompt prefix | The full response body |
| What's saved | Prefill compute + cost on the prefix | The entire computation |
| Output | Generated fresh every call | Returned identical from cache |
| Hit condition | Identical leading prefix | Identical full request/key |
Real products, models, and research that use this idea.
- Anthropic prompt caching lets you mark a stable prefix with cache_control and bills cached reads at a fraction of input price.
- OpenAI automatic prompt caching reuses repeated prompt prefixes above a token threshold and discounts the cached input.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow should you order a prompt to maximize cache hits?
Put stable content (system prompt, shared document) first and volatile content (query, timestamp, IDs) last.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking caching stores the model's answer like an HTTP cache — it stores the KV state of the prompt prefix, so it speeds prefill but still generates a fresh output every call.
60 second bullets to scan on the way to the call.
What the cache actually stores: KV state of a leading prefix
Which phase it accelerates and why output is unaffected
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.