Explain what provider prompt (prefix) caching actually caches and the workload shape that makes it pay off.
Free · 2 AI evals / day
When an LLM reads your prompt, it does a pass called **prefill** that turns every prompt token into internal state — the KV cache — before it writes a single word of the answer. Prefill is real work, and you pay for it in time and money.
Prefix caching saves that work for the part of the prompt that doesn't change. If the start of your prompt is identical across many requests — a long system prompt, a shared document, a fixed set of examples — the provider can compute its KV state once and reuse it. Later requests with the same opening skip re-doing prefill over it.
Two things to keep straight. It caches the prompt's internal state, not the answer — every call still generates fresh output. And providers bill those cached tokens at a steep discount.
It pays off when lots of requests share a big, unchanging head: a chatbot with a large fixed system prompt, RAG over a reused document, repeated few-shot prompts. It does almost nothing if prompts are short or every request starts differently.
Prefix caching stores the computed **KV-cache state** of a stable leading segment of the prompt. The first time a prompt is seen, the provider runs prefill over the whole thing; it retains the key/value tensors for the leading prefix keyed on its exact token content. Subsequent requests that begin with the byte for byte identical prefix reuse that state and run prefill only over the new suffix.
The saving is on prefill — compute and the latency it adds to time to first token. It is also on cost: cached input tokens bill at a large discount, often a small fraction of the normal input rate.
The requirement is what makes or breaks it. The cache keys on an **exact, stable prefix**, so it fires only when many requests share an identical leading segment. That's why prompt layout matters: stable content (system prompt, shared document, few-shot examples) goes at the front; anything volatile (the live user query, timestamps, IDs) goes after it.
The workloads that benefit most all share one trait. They have a large, reused head. Chat applications prepend a big fixed system prompt every turn. RAG or document QA queries one large document many times. Few-shot pipelines prepend the same example block to every call.
It does little when prompts are short — there's no expensive prefix to amortize — or when every request's prefix differs, so the cache never hits.
Quick reference
TL;DR
Prefix caching stores the KV state of a stable leading prompt segment so repeat calls skip prefill over it; it pays off when many requests share a large, identical head.
Imagine a teacher who reads the same long set of class rules aloud before every lesson. Instead of re-reading the whole thing each time, she records it once and just plays the recording, then adds the day's new material live.
Prompt caching does that for an LLM. The long, unchanging part at the start of your prompt — the standing instructions, a shared document — gets processed once and saved. Later requests that begin with the exact same opening skip re-processing it and pick up from where the saved part ends. You save time and money on the repeated part, but only if it stays word for word the same and sits at the very front.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Prompts where the leading content changes on every request No stable prefix means the cache never hits, so you pay full prefill plus any cache-write overhead.
Short prompts under a few hundred tokens There's no expensive prefix to amortize, and providers often won't cache below a token threshold anyway.
Sparse, low-frequency traffic on a given prefix Cache entries expire after a short TTL, so infrequent calls miss and re-pay prefill, negating the win.
Quick scan · 2–4 min
Everything important, quickly.
Spend 6-8 minutes on what gets cached (KV state of the prefix), the exact-prefix requirement, the workloads that benefit, and the cost discount.
Side-by-side
Aspect
Prefix caching
HTTP-style response cache
What's stored
KV state of the prompt prefix
The full response body
What's saved
Prefill compute + cost on the prefix
The entire computation
Output
Generated fresh every call
Returned identical from cache
Hit condition
Identical leading prefix
Identical full request/key
Real-world examples
Real products, models, and research that use this idea.
1Anthropic prompt caching lets you mark a stable prefix with cache_control and bills cached reads at a fraction of input price.
2OpenAI automatic prompt caching reuses repeated prompt prefixes above a token threshold and discounts the cached input.
3RAG and agent frameworks place a large shared document or tool preamble first so the prefix stays cacheable across queries.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow should you order a prompt to maximize cache hits?
Sign in or go Pro to see the approach for this follow-up.
QWhy does caching help TTFT but not the streaming speed of the answer?
Sign in or go Pro to see the approach for this follow-up.
QWhen does the cache-write premium make caching a net loss?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking caching stores the model's answer like an HTTP cache — it stores the KV state of the prompt prefix, so it speeds prefill but still generates a fresh output every call.