The cacheable portion is the stable prefix from start of prompt up to the last byte-identical point across calls; the variable user query at the tail is not cached.
Imagine ordering at the same restaurant every day. The first time, the chef has to read the whole menu, prep the kitchen, and learn your preferences. Every day after, they remember the menu, the kitchen is ready, and they only need to hear what you want today. Prompt caching works the same way for an LLM. The system prompt and the long retrieved context are like the menu and the kitchen prep. They stay the same call after call, so the provider keeps them ready and you pay almost nothing for them on the next call. The new user query is the only fresh part the model has to process from scratch.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt caching is the single highest-leverage cost optimization for production LLM apps that share long stable prefixes across calls. The mechanism is straightforward: the provider keeps the attention state for a prefix the model has already processed and reuses it on the next call whose prefix is byte-identical. The pricing is dramatically discounted on the cached portion, and the savings compound across traffic.
The goal of this deep dive is to give a clear mental model of where the cache boundary sits in a typical RAG or chat prompt, how Anthropic and OpenAI differ in cache mechanics and pricing, and what the operational failure modes look like. The interview-worthy framing is to identify the stable prefix, name the boundary, and call out the byte-exact gotcha that silently kills hit rate in production.
The sections below walk through KV-cache reuse as the underlying mechanism, the provider-side differences, the stability-ordered prompt layout that maximizes hit rate, and the failure modes a senior engineer should be able to diagnose from cache telemetry.
KV-cache reuse as the underlying mechanism
Every LLM inference call materializes a key-value cache as it processes the prompt. The KV cache is the attention state for each token: the keys and values produced by each layer that future tokens will attend over. For an input prefix of length N, the model does N forward passes to build the KV cache before it can start generating tokens.
Prompt caching exposes that KV state at the API layer. The provider hashes the input prefix, stores the materialized KV cache against the hash, and on a subsequent call with the same hash, reuses the stored state instead of recomputing it. The cost savings come from skipping the prefill compute on the cached portion.
The key invariant is that the cache key is the exact byte sequence of the prefix. The hash is over raw tokens (or raw bytes upstream of tokenization). Any change to the prefix changes the hash, and a different hash means a fresh prefill. This is what makes byte-exact match the binding constraint on hit rate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt caching for Claude Opus 4.7 prices cached input at one tenth of normal cost, cutting agentic loop spend dramatically on stable system prompts plus tool definitions.
- OpenAI's automatic prefix caching for GPT-5.5 fires on prefixes of 1024 tokens or more at roughly half price, with no explicit markers required.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the right prompt ordering to maximize hit rate?
Stability-ordered from top to bottom: system and principles first, tool definitions and long-lived examples next, retrieved context, then current user state and query at the bottom; mark the cache boundary at the last stable point.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the cache fires on similar prompts when it requires a byte-identical prefix; even a session id injected at the top can knock the cache out.
60 second bullets to scan on the way to the call.
Where the cache boundary sits in a typical RAG prompt
Anthropic vs OpenAI cached-token pricing
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.