Prompt Caching
Also known as: Provider prompt caching
Provider caches the attention state of common prompt prefixes, so the same prefix in next request skips that compute.
A provider-side optimization where common prompt prefixes (system prompts, RAG context blocks) are cached server-side so subsequent requests with the same prefix skip redundant attention compute. Anthropic and OpenAI both expose this.
In practice
Direct cost reduction (75-90% off on the cached portion). Knowing what prefixes to cache and TTL constraints matters for cost-tuning interviews.
How it compares
KV cache reuses attention within a single generation; prompt caching reuses that cache across separate requests that share a prefix.
Prompt caching skips compute for byte-identical prefixes; semantic cache skips the whole inference call for semantically similar prompts.
Comparisons that include Prompt Caching
Related topics
Questions that mention this term
- Decompose the cost of a single API call into its components and explain which dominates.
- Walk through the prompt structure that maximizes provider prompt cache hit rate
- Pick the metric that best signals you are getting value from provider prompt caching
- Why scan user input…
- Fill in the Anthropic prompt caching primitive and its typical discount
- Order the cache layers an LLM request should check before paying for tokens
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.