Explain how prompt caching is priced by Anthropic and by OpenAI. What counts as a cache hit, what doesn't, and what should an application architect do to maximize hit rate?
Free · 2 AI evals / day
**Prompt caching** lets a provider remember the work it did on the front of your prompt so it does not redo it next time. Internally this is the KV cache from the prefill phase, kept warm and reused.
Anthropic charges cached input tokens at roughly 10 percent of the normal input rate. There is a one-time write surcharge (about 25 percent above input) the first time an entry is stored. The cache expires after about 5 minutes by default, with longer options available.
OpenAI takes a different shape. It gives a flat 50 percent discount on cached input tokens, charges no explicit write fee, and turns caching on automatically once the prompt passes a length threshold.
A cache hit is strict. The bytes of the cached prefix must exactly match a previous request. Change the system prompt, the tool schema, or even leading whitespace, and the hit breaks for everything after that point.
The design rule follows directly: put long stable content (system prompt, documents, few-shot examples, tools) first, and the changing user query last. Then the largest part of every request reuses the cache.
**Prompt caching** stores the prefill KV state for a prompt prefix and replays it at a discount, so the model skips recomputing tokens it has already processed.
The two providers price it differently. Anthropic charges cached reads at about 10 percent of the input rate. It adds a one-time write surcharge near 25 percent above input when a new entry is stored. The default cache lives about 5 minutes, with longer tiers available. OpenAI applies a flat 50 percent discount on cached input. It charges no separate write fee and caches automatically once a prompt clears a length threshold.
A cache hit is a strict byte-equality check on the prefix. Any change above the dynamic suffix breaks it. That includes the system prompt, the tool schema, few-shot examples, even leading whitespace or JSON key order. There is no semantic matching.
The architecture rule follows. Place the largest stable content at the front. Put the volatile user query last. Then the cacheable prefix dominates each request. Hit rate, not raw token count, drives per-call cost on agents, retrieval pipelines, and support bots that share long context across many turns.
Quick reference
TL;DR
Prompt caching reuses the KV state of a byte-identical prefix at a steep discount, so put stable content first and dynamic content last to maximize hit rate.
Imagine a coffee shop where the barista has to read your whole life story before every order, charging you per word. With prompt caching, the barista writes your fixed story (who you are, your usual notes) onto a card once, keeps it behind the counter, and only reads the new bit each visit. Reading the card costs almost nothing compared to re-reading everything. But the card only works if the story is word for word identical to last time. Change one comma at the top and the barista throws the card away and rereads it all. The card also expires after a few minutes of no visits. So you keep the unchanging part at the top, the new order at the bottom, and you visit often enough that the card stays warm.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
One-shot prompts that are never repeated within the cache TTL window On Anthropic the write surcharge is never amortized, so caching costs more than a plain uncached call.
Prompts whose prefix legitimately changes on every request If the stable portion is tiny, the cacheable prefix is too small to move per-call cost meaningfully.
Latency-critical paths where the prefix is short and already cheap The savings are negligible and the added prompt-ordering complexity is not worth the engineering cost.
Sparse low-traffic endpoints with long gaps between calls The cache expires between requests, so most calls miss and pay full price plus repeated write fees.
Quick scan · 2–4 min
Everything important, quickly.
4 min: byte-identical prefix hit rule + Anthropic 10 percent read, write fee, TTL + OpenAI flat 50 percent auto + prompt ordering rule + savings math + high-impact workloads.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's Claude Opus 4.7 API exposes cache_control breakpoints so an agent can cache a long system prompt and tool schema across a multi-turn loop.
2OpenAI's GPT-5.5 automatically caches prompt prefixes above its length threshold, giving a 50 percent input discount with no code changes.
3RAG stacks on LangChain place retrieved documents and few-shot examples ahead of the user query specifically to keep the cached prefix large.
4Cursor and similar coding agents cache a stable repository-context and instruction header so each edit request only pays full price on the new diff.
5Customer support bots cache a shared policy and persona system prompt across thousands of sessions, where the prefix is the bulk of every request.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Anthropic add a write surcharge while OpenAI does not?
Sign in or go Pro to see the approach for this follow-up.
QHow would you measure and monitor cache hit rate in production?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the relationship between prompt caching and the underlying KV cache?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming any semantically similar prompt hits the cache. A hit needs a byte-identical prefix, so one changed token, reordered tool, or stray whitespace high in the prompt invalidates everything below it.