KV Cache
Also known as: Key-value cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
A memory optimization in transformer inference that stores previously computed key-value pairs from self-attention layers, avoiding redundant recomputation during autoregressive token generation.
In practice
The single biggest reason long-context inference is feasible. Expect questions on cache memory math (layers × heads × dim × seqlen × 2) and paged-attention variants.
Comparisons that include KV Cache
Related topics
Practice questions
Questions that mention this term
- Llama-2 70B uses 64 query heads, how many KV heads does it actually keep?
- Match each long context strategy to what it modifies.
- Why does autoregressive generation use a KV cache?
- INT8 KV cache vs INT8 weight quantization, which one is easier in production?
- Match each attention variant…
- Translating one source sentence into many target languages, what attention side trick scales?
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.