Prompt caches only hit on byte-stable prefixes, so the system prompt should hold turn-invariant content (persona, policies, tool schemas) and never per-call values like timestamps or user ids.
Imagine a coffee shop that pre-makes your standard order the moment you walk in, but only if you ask for the exact same thing every time. Change one word, say, oat milk instead of whole milk, and the barista has to start from scratch. The shop's pre-make station is the prompt cache. Your standard order is everything in the system prompt that never changes: who the model is, what it can do, what shape its answer should take. If you sneak today's date or your username into that standard order, the barista throws out the half-made drink and starts over. The trick is to keep your standard order genuinely standard and ask for the day's specials at the counter instead.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching is one of the cleanest cost wins in 2026 LLM operations, and it is also one of the easiest to silently break. Anthropic, OpenAI, and Google all expose some form of it: a way to mark a prefix of the prompt as cacheable, after which subsequent calls with the same prefix reuse the model's internal state at a fraction of the cost and latency.
The feature only works on byte-exact prefixes. That sounds obvious, but in practice the discipline of keeping a multi thousand token system prompt truly byte-stable across calls is harder than it looks. This deep dive walks through what belongs in the system prompt under that constraint, what does not, how to layer prompts across multiple cache breakpoints, and how to detect regressions before they show up on the invoice.
What gets cached and what the cache key is
Prompt caching reuses the model's per-token internal state, the result of the forward pass over the cached prefix, keyed by the literal byte sequence of that prefix. When a new request shares the same prefix bytes, the model skips the recomputation and starts from the boundary of the cached region. Anthropic bills cached tokens at roughly one-tenth the standard input rate after a one-time cache write cost. OpenAI's caching is automatic for prefixes of 1024+ tokens on supported models and bills at a 50-75% discount. Google's Gemini context caching is explicit, with a user-defined TTL.
The critical property is byte-exactness. The cache does not match on semantics,"the same persona, slightly reworded" is a cache miss. It does not match on structure,"the same tool schema with the keys in a different order" is also a miss. It matches on the literal sequence of bytes from position zero to the cache breakpoint.
This means that anything that can vary between requests must live outside the cached span. The corollary is that the design of the system prompt is no longer just a content question; it is a stability question. Every line in the system prompt is implicitly making a promise that this line will be the same on the next request, and the request after that, and so on for the lifetime of the prompt version.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude API exposes explicit cache_control breakpoints; production teams put persona, tool schemas, and policy in cached blocks while keeping the user turn fresh.
- OpenAI's Responses API auto-caches stable prefixes on the GPT-5 and GPT-5.5 family; teams cache long retrieval-policy preambles and pay only for the per-turn tail.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you design a long prompt to maximize cache hits when the prompt has multiple layers of stability?
Lay the prompt out in concentric rings: globally stable persona and policies first, then long stable tool schemas, then per-tenant policy, then per-turn variables. Place cache breakpoints between rings so each tier caches at its own rate. The Anthropic API allows up to four breakpoints, which is enough for most production layouts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Sprinkling today's date, the active user id, or feature flags into the system prompt and silently destroying every cache hit on the application.
60 second bullets to scan on the way to the call.
State the byte-exact property of the cache key and what it implies
List the categories of content that belong in the system prompt
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.