Prompt caching stores the KV state of a prompt prefix server-side so future requests sharing that prefix skip the prefill compute, dropping both cost (cached input billed at a steep discount) and time to first token.
Imagine a coffee shop where every customer asks for the same long list of background details before placing their actual order: 'Hi, I am a tall person, I drink mostly black coffee, I am allergic to dairy, I prefer ceramic cups, I'm in a hurry today, AND I would like a flat white.' The barista has to listen to the whole script every single time. Now imagine the shop remembers your background details from yesterday. You walk in and just say 'flat white' and the barista already knows the rest. The shop charges you a small fee to remember your details, but each subsequent order is faster and cheaper because they skip re-listening to the script. That is prompt caching. The repeated prefix is the script. Skipping the re-read is the savings.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching is the productized form of an inference-engineering observation: prefill is expensive, and most production prompts share long prefixes. A chat app sends the same system prompt every turn. An agent framework includes the same tool definitions on every call. A RAG pipeline re-sends the same retrieved chunks until the user asks a different question. If the inference provider can recognize these repeated prefixes and reuse the work, both cost and latency drop dramatically.
This deep dive walks through what prompt caching actually does inside the server, how the pricing structures of Anthropic and OpenAI map to the underlying mechanism, the prompt-structure discipline required to hit the cache reliably, and the self-hosted equivalents available in vLLM and SGLang. By the end, you should be able to look at any LLM application's prompt template and predict its cache hit rate.
What prompt caching actually does
Recall the structure of LLM inference: prefill processes the prompt and populates the KV cache, then decode generates output tokens by repeatedly consulting that cache. Prefill is the expensive part: it scales with prompt length and dominates time to first token. For a 10,000-token system prompt, prefill might take a full second of compute on H100-class hardware.
Prompt caching observes that if the same prefix appears at the start of many requests, the KV cache state for that prefix is identical every time. So the provider can compute it once, store it in HBM (or fall back to SSD), and reuse it on future requests. When a new request comes in:
- The provider tokenizes the prompt.
- It hashes the prefix incrementally and checks against the cache index.
- If a match is found, it loads the cached KV state directly into the inference engine.
- Fresh prefill only runs on the tokens that appear after the cached portion.
- Decode proceeds normally from the (cached prefix + fresh tokens) combined KV state.
The match must be an exact tokenized prefix. A single character change near the start of the prompt produces a different tokenization (or at minimum a different token at that position), which means the hash changes, and subsequent tokens' KV values depend on the changed token through self-attention. The cache is invalidated from that point forward.
This is why prompt caching is sometimes called prefix caching: the unit of reuse is always a contiguous run of tokens from the beginning of the prompt. You cannot cache a chunk from the middle; the KV state of position i depends on every position < i through the attention mechanism.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Anthropic | OpenAI | Self-hosted (vLLM, SGLang) |
|---|---|---|---|
| Discount on cached reads | ~90% off | ~50% off | Free (you own the GPU) |
| Write surcharge | ~25% on first call | None | None |
| Activation | Explicit cache_control markers | Automatic for prefixes >=1024 tokens | Configurable feature flag |
| Typical TTL | 5 minutes default (extended tiers available) | 5-10 minutes | Configurable; bounded by GPU/SSD |
| Cache key | Tokenized prefix exact match | Tokenized prefix exact match | Tokenized prefix; tree-structured in SGLang |
Real products, models, and research that use this idea.
- Claude Opus 4.7 with a 10000-token tool-definitions system prompt sees TTFT drop from ~1500ms to ~200ms when prompt caching is enabled on Anthropic's API.
- OpenAI's GPT-5.5 automatically caches prefixes of at least 1024 tokens, discounting them at 50% on subsequent matching calls within the TTL.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does Anthropic charge a write surcharge while OpenAI does not?
Anthropic exposes explicit cache_control markers, so users opt in and the surcharge incentivizes thoughtful placement. OpenAI's automatic detection avoids charging for caches the user did not request. Both pricing models can be self-consistent given their UX choice.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking prompt caching saves on output tokens. It only discounts input (prompt) tokens that match a cached prefix; output token billing is unchanged.
60 second bullets to scan on the way to the call.
Definition of prompt caching as skipping prefill for repeated prefixes
Which phase of inference it accelerates and which it does not affect
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.