Prompt caching shows up on Anthropic and OpenAI pricing pages, what does it actually do?

Attempt it

Prompt caching is a provider-side feature that stores a hashed prefix of a prompt so repeated calls that start with the same prefix can skip the prefill compute for that prefix. Cached input tokens are billed at a steep discount (Anthropic ~90% off cached reads with a ~25% write surcharge, OpenAI ~50% off automatic for prefixes >=1024 tokens) and time to first token drops because the heavy prefill work is reused instead of redone.

Your answer

Quick reference

TL;DR

Prompt caching stores the KV state of a prompt prefix server-side so future requests sharing that prefix skip the prefill compute, dropping both cost (cached input billed at a steep discount) and time to first token.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a coffee shop where every customer asks for the same long list of background details before placing their actual order: 'Hi, I am a tall person, I drink mostly black coffee, I am allergic to dairy, I prefer ceramic cups, I'm in a hurry today, AND I would like a flat white.' The barista has to listen to the whole script every single time. Now imagine the shop remembers your background details from yesterday. You walk in and just say 'flat white' and the barista already knows the rest. The shop charges you a small fee to remember your details, but each subsequent order is faster and cheaper because they skip re-listening to the script. That is prompt caching. The repeated prefix is the script. Skipping the re-read is the savings.

Key concepts

Prompt caching Prefill phase KV cacheTime to first token

Quick scan · 2–4 min

Everything important, quickly.

3 min: definition + which phase it skips + Anthropic and OpenAI pricing + structure the prompt rule (stable first, variable last) + TTL constraint + self-hosted equivalents.

Side-by-side

Aspect	Anthropic	OpenAI	Self-hosted (vLLM, SGLang)
Discount on cached reads	~90% off	~50% off	Free (you own the GPU)
Write surcharge	~25% on first call	None	None
Activation	Explicit cache_control markers	Automatic for prefixes >=1024 tokens	Configurable feature flag
Typical TTL	5 minutes default (extended tiers available)	5-10 minutes	Configurable; bounded by GPU/SSD
Cache key	Tokenized prefix exact match	Tokenized prefix exact match	Tokenized prefix; tree-structured in SGLang

Real-world examples

Real products, models, and research that use this idea.

Claude Opus 4.7 with a 10000-token tool-definitions system prompt sees TTFT drop from ~1500ms to ~200ms when prompt caching is enabled on Anthropic's API.
OpenAI's GPT-5.5 automatically caches prefixes of at least 1024 tokens, discounting them at 50% on subsequent matching calls within the TTL.
vLLM's prefix caching feature applies the same idea to self-hosted inference, deduplicating KV cache across requests that share a system prompt.
SGLang's RadixAttention extends prefix caching to a tree structure, so concurrent requests sharing partial prefixes can all benefit from cached KV state.
Cursor and Claude Code use Anthropic's prompt caching to bring per-edit latency down by reusing the cached tool-definitions and codebase context across calls.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does Anthropic charge a write surcharge while OpenAI does not?

Sign in or go Pro to see the approach for this follow-up.

QHow does prefix caching in vLLM compare to RadixAttention in SGLang?

Sign in or go Pro to see the approach for this follow-up.

QWhat happens to the cache when you change a single token near the start of the prompt?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.