What does the KV cache store, and why does it avoid Q?
The KV cache stores K and V for every past token at every layer and head; Q isn't cached because each token's Q is only used at its own step and never re-read later.
Imagine a long group conversation where you keep one sticky note per person summarizing what they said and what they care about. When a new person speaks, they glance at every sticky note to decide what to say, but the fleeting thought that prompted their own question evaporates the moment they finish speaking. The sticky notes are like K and V (read again and again by future speakers); the fleeting thought is like Q (used once, then gone).
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5m: causal attention directionality, what is and isn't cached, the per-token memory formula, prefill vs decode bottlenecks, and the GQA/MQA/MLA + PagedAttention chain that follows from cache pressure.
# Conceptual decode step with a KV cache.
# past_k, past_v: [batch, n_kv_heads, seq_so_far, d_head]
# x_new: hidden state of the new token, shape [batch, 1, d_model]
def decode_step(x_new, past_k, past_v, W_q, W_k, W_v, W_o):
q_new = x_new @ W_q # [B, 1, n_heads * d_head]
k_new = x_new @ W_k # [B, 1, n_kv_heads * d_head]
v_new = x_new @ W_v # [B, 1, n_kv_heads * d_head]
# Append new K, V to the cache. No past Q is stored.
k = concat([past_k, reshape(k_new)], dim=2)
v = concat([past_v, reshape(v_new)], dim=2)
# Attention: only the new Q against the full cached K, V.
scores = (q_new @ k.transpose(-1, -2)) / sqrt(d_head)
attn = softmax(scores, dim=-1)
out = attn @ v
return out @ W_o, k, v # return updated cacheReal products, models, and research that use this idea.
- HuggingFace transformers exposes the cache via past_key_values; you pass it in and receive an updated cache out at each generation step.
- vLLM's PagedAttention is a production KV-cache allocator using OS-style paging to eliminate fragmentation and unlock multi-tenant prefix sharing.
- Llama 4 Maverick, Claude Opus 4.7, and Gemini 3.1 Pro all use grouped query attention to shrink the KV footprint relative to vanilla multi-head attention.
- DeepSeek V4 uses Multi-head Latent Attention, compressing the cache into a low rank latent space and decompressing per step for further memory savings.
- llama.cpp keeps the cache in a contiguous buffer and supports KV quantization formats such as Q8_0 and Q4_0 for memory constrained deployments.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the cache differ between prefill and decode in a production serving stack?
QWhat does the cache look like across a batch of requests with very different lengths?
QWhy doesn't MLA simply replace GQA everywhere if it shrinks the cache further?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating the cache as a generic 'attention cache' that also holds Q, weights, or outputs. Only the K and V tensors are persisted across steps; everything else is recomputed or discarded.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.