Prompt caching only pays off when the cached prefix is stable across calls. The system prompt is therefore the right home for content that is identical from turn to turn: persona, policies, output format contract, tool definitions. The moment any per-call value bleeds in, a timestamp, the current user id, a dynamic feature flag, today's date, the cache key changes and the cache miss eats the savings. The discipline is to keep the system prompt byte-stable across calls, and inject anything per-turn into a later, uncached part of the prompt (a user-turn header or a developer message).
**Prompt caching** is a feature offered by Anthropic, OpenAI, and Google that lets you reuse the static prefix of a prompt across many calls without paying full input-token cost or full latency on the cached portion. It works by hashing the prefix and reusing the model's internal state when the same prefix shows up again.
The catch is that caching is byte-exact. The cache key is the literal text of the prefix, so even a one-character change invalidates the entry and the next call pays the full cost again.
This directly shapes what belongs in the system prompt. Content that is identical from turn to turn fits naturally: the assistant's persona, the rules it must follow, the tool definitions, and the output format contract. All of that is stable across users and across turns.
What does NOT belong in the system prompt is anything that varies per call: the current date, the user's id, a session-specific flag, the active locale, or a personalization token. The moment one of those bleeds into the system block, the cache key shifts on every request and the application gets a 100% miss rate.
The fix is to move per-turn values out of the system block into a later, uncached message, a user-turn header, a developer message, or the first line of the user content.
**Prompt caching** reuses the model's internal computation over a stable prefix of the prompt, billing those tokens at a steep discount (typically 10x cheaper input cost and a meaningful latency cut). Anthropic, OpenAI, and Google all ship a version of this feature in 2026.
The critical property is that the cache key is byte-exact over the prefix. Any change inside the cached span, a different timestamp, a different user id, a swapped feature flag, invalidates the entry and the next call rebuilds from scratch.
This turns the system prompt into a discipline problem rather than a free-text scratchpad. Content that is genuinely stable across calls belongs there: persona description, behavioral rules, safety policies, output format contract, and tool / function definitions. These do not vary turn to turn, so they cache cleanly.
Content that varies per call must move out: current date or time, the active user's id or profile, session-specific feature flags, dynamic locale, request id, A/B-test bucket. The right home for these is later in the request, in a developer message or in the first lines of the user message, where the cache does not apply but the cost is only one short block instead of the entire prompt.
The operational rule is simple. Treat the system prompt as a published artifact that changes when the product changes, not when the request changes. Every per-call value gets injected after the cached prefix ends.
Quick reference
TL;DR
Prompt caches only hit on byte-stable prefixes, so the system prompt should hold turn-invariant content (persona, policies, tool schemas) and never per-call values like timestamps or user ids.
Imagine a coffee shop that pre-makes your standard order the moment you walk in, but only if you ask for the exact same thing every time. Change one word, say, oat milk instead of whole milk, and the barista has to start from scratch. The shop's pre-make station is the prompt cache. Your standard order is everything in the system prompt that never changes: who the model is, what it can do, what shape its answer should take. If you sneak today's date or your username into that standard order, the barista throws out the half-made drink and starts over. The trick is to keep your standard order genuinely standard and ask for the day's specials at the counter instead.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Prompts that are genuinely unique per call, with no shared prefix worth caching. If every call has a different long input, there is nothing for the cache to amortize and the overhead of declaring breakpoints is not repaid.
Workloads where the system prompt is small (under a few hundred tokens). Caching has a minimum-token threshold (Anthropic requires 1024 tokens for Sonnet/Opus). Below that, the cache will not apply and the design effort is wasted.
Experiments that intentionally vary the system prompt to evaluate behavior changes. Caching makes a virtue of stability, which is the opposite of what an evaluation sweep wants. Disable cache breakpoints during prompt ablation runs.
Quick scan · 2–4 min
Everything important, quickly.
Define prompt caching and the byte-exact cache key. State the rule: only turn-invariant content in the system prompt. List what belongs (persona, policies, output contract, tool schemas) and what does not (timestamps, user id, feature flags, locale). Explain where per-call values go instead. Touch on cache breakpoints for layered prompts and the failure modes (template whitespace, dict ordering, build-time interpolations). Close with the observability point: surface cache-hit rate or you will not notice regressions.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's Claude API exposes explicit cache_control breakpoints; production teams put persona, tool schemas, and policy in cached blocks while keeping the user turn fresh.
2OpenAI's Responses API auto-caches stable prefixes on the GPT-5 and GPT-5.5 family; teams cache long retrieval-policy preambles and pay only for the per-turn tail.
3Cursor and Claude Code keep multi thousand token coding-agent system prompts byte-stable across sessions to maximize cache hits on every editor command.
4LangChain and LlamaIndex have shipped cache-aware prompt builders that warn when a per-turn variable lands inside a declared cached span.
5Vercel AI SDK exposes a providerOptions.anthropic.cacheControl helper so React Server Components can declare which prompt sections are cacheable.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you design a long prompt to maximize cache hits when the prompt has multiple layers of stability?
Sign in or go Pro to see the approach for this follow-up.
QWhat invisible sources of non-determinism break prompt caching even when the visible prompt looks identical?
Sign in or go Pro to see the approach for this follow-up.
QHow does prompt caching interact with safety and personalization at the same time?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Sprinkling today's date, the active user id, or feature flags into the system prompt and silently destroying every cache hit on the application.