Caches key on exact token ids, not meaning. One changed word shifts ids from that point forward, invalidating every KV tensor after it because attention at position N depends on all earlier positions.
Imagine a long shared road the model drives down for every request. Once the model has driven a mile, the road behind has cone markers showing it has been measured. The next driver can skip the measured section. Now suppose someone moves a single cone earlier in the road. Every cone past that point is suspicious; the next driver has to re-measure from the moved cone onward. The cones are the cached KV values, the road is the token prefix, and moving a cone is what happens when you change a single character in the prompt.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching is one of the highest leverage cost optimizations in 2026 LLM deployment, but it only works when the caller respects the cache's contract. That contract is at the token-id level, not the character level, and the gap between what looks the same to a human and what the cache treats as the same is where most missed hits live.
This explanation walks through the mechanics, why alignment is at the token-id level, how each major provider exposes the feature, and the design patterns that keep cache hit rate high.
How prompt caching works under the hood
When the model processes a sequence, it produces K and V tensors for every position. These tensors are exactly what attention at later positions needs. Normally they are computed fresh every request and discarded.
Prompt caching changes that. The serving engine stores the K and V tensors for the prefix and indexes them by the token-id sequence of that prefix. When a new request arrives, the engine walks the new request's token ids against cached prefixes and finds the longest match. The cached K and V for the matching prefix are reused; only tokens past the match point need recomputation.
The savings come from skipping prefill for the matched portion. For a long prompt with a short per-call addition, prefill is the dominant cost. A 50K token shared system prompt with a 500 token user message can see the cache eliminate 99% of prefill work.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic Claude API: cache_control markers in message blocks; cached input billed at roughly one-tenth of normal input price in 2026.
- OpenAI prompt caching: automatic for prompts above a minimum token length on the GPT-5.5 family; usage.prompt_tokens_details.cached_tokens reports hits.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf I edit a single word in the middle of a 10K token system prompt, how much of the cache is invalidated?
Everything from the first token where the new ids differ to the end. If the edit is at token position 4,000 of 10,000, then 6,000 tokens recompute and re-cache. The earlier 4,000 are still hits.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Putting dynamic content (user question, timestamp) at the start of the prompt and the long static system prompt at the end. The first token differs every call, so the cache hit rate drops to zero.
60 second bullets to scan on the way to the call.
Define prompt caching as KV-tensor reuse keyed on exact token-id prefix match.
Explain why the cache is at the token-id level rather than character level.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.