Prompt Caching
Also known as: Provider prompt caching
Provider caches the attention state of common prompt prefixes, so the same prefix in next request skips that compute.
A provider-side optimization where common prompt prefixes (system prompts, RAG context blocks) are cached server-side so subsequent requests with the same prefix skip redundant attention compute. Anthropic and OpenAI both expose this.
In practice
Direct cost reduction (75-90% off on the cached portion). Knowing what prefixes to cache and TTL constraints matters for cost-tuning interviews.
How it compares
KV cache reuses attention within a single generation; prompt caching reuses that cache across separate requests that share a prefix.
Prompt caching skips compute for byte-identical prefixes; semantic cache skips the whole inference call for semantically similar prompts.
Comparisons that include Prompt Caching
Related topics
Related terms
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
Knowledge Distillation
Train a small student model to match a big teacher's outputs: cheap, fast inference with most of the quality.