Prompt caching vs KV caching
Two ways to stop paying for the same tokens twice
KV cache reuses attention within a single generation; prompt caching reuses that cache across separate requests that share a prefix. Both attack the same waste from different angles.
KV cache
Glossary →During autoregressive generation, the model caches the key and value tensors it already computed for prior tokens so it does not recompute them for each next-token step. Built into every transformer inference stack.
Best for: Every single LLM generation, always.
Prompt caching
Glossary →A provider feature (Anthropic, OpenAI, Google) that stores the KV cache for a stable prompt prefix and reuses it across separate requests. Cached input tokens bill at a large discount (usually 50 to 90 percent) and first-token latency drops.
Best for: Long stable system prompts reused across many calls.
At a glance
| Dimension | KV cache | Prompt caching |
|---|---|---|
| Scope | Within one generation | Across separate requests |
| Where it lives | In the inference process | Provider or shared server |
| Requires code change | No (built in) | Yes (mark the cacheable prefix) |
| First-token latency | Small, per token | Drops sharply on cache hit |
| Cost impact | Compute only | Compute plus input-token discount (50-90%) |
| Best for | Every LLM call | Long stable prefixes (system prompt, tool defs) |
Key differences
- 1KV cache lives inside one generation; prompt caching lives across many requests
- 2KV cache is universal in transformer inference; prompt caching is a provider (or serving-framework) feature
- 3KV cache never sees the network; prompt caching hands the cache to the provider or a shared server
- 4KV cache saves compute; prompt caching saves both compute and input-token bill
- 5You always benefit from KV cache; you only benefit from prompt caching when a prefix is stable across requests
In the interview
- Conflating KV cache with prompt caching
- Claiming prompt caching makes prompts free (input tokens are discounted, not zero)
- Ignoring the provider-side prefix invalidation rules (any change busts the cache)
- Missing that shared serving frameworks (vLLM, TGI) can implement similar prefix reuse locally
How to choose
Always use the KV cache. Turn on prompt caching when the shared prefix is a big fraction of the input.
Common misconceptions
Myth: Prompt caching removes the need for cheap prompts.
Reality: Cached input tokens still bill, just at a large discount. A long stable system prompt with prompt caching is still worth trimming.
Myth: KV cache saves cost across users.
Reality: Standard KV cache is per-generation only. Cross-request reuse needs a serving framework feature or the provider's prompt-caching API.
Memory aid
KV cache = remembering what you already computed for THIS answer. Prompt caching = not re-reading the same instructions every time a new question arrives.
Can you combine them?
They stack: KV cache runs inside every generation; prompt caching reuses the cache across requests with the same prefix. Together they turn a long shared system prompt from a per-request cost into a build-once, reuse-many artifact.
Related topics
Related comparisons
API vs Self-hosted LLMs
Rent the best model, or run your own
Distillation vs Quantization
Two orthogonal ways to shrink a large language model
Greedy vs Beam Search vs Sampling
Three ways to decode tokens from a language model