Explain what the KV cache stores during autoregressive decoding and what work it lets you skip on each new token step. Be specific about which tensors are cached and which are recomputed.
Free · 2 AI evals / day
In a transformer, each token is projected into three vectors: a query Q, a key K, and a value V. The attention output for a token is a weighted sum of every value, where the weights come from that token's query dotted against every key.
When the model generates text, it emits one token at a time. To produce the next token, its attention layers need the keys and values of every token that came before. A naive decoder would re-project all of those past tokens into K and V on every single step, which is wasteful because those projections never change.
The insight is that a past token's K and V are fixed functions of that token's hidden state, and that hidden state is frozen once the token is generated. So you compute each token's K and V once and keep them.
The **KV cache** is that storage, organised by layer, by head, and by token position. On each decode step you compute K and V only for the one new token, append them, and read the whole cache to do attention. Q for past tokens is never kept, because no later step ever looks at it again.
**The KV cache** stores the key and value projections of every token already processed, indexed by layer, head, and position. It exists so autoregressive decoding can attend to the full history without re-projecting the prefix each step.
Mechanism. At each decode step the model computes Q, K, and V for the single new token only. It appends that token's K and V to the cache. The new query then dot-products against all cached keys, softmaxes, and forms a weighted sum over all cached values.
The asymmetry is the crux. K and V are cached because every future token will attend to them. Q is recomputed fresh and never stored, because a past token's query is consumed once during its own step and never read again.
This is what collapses the cost. Without the cache, regenerating K and V for the whole prefix at every step is O(N) work per step, so O(N^2) over a full completion. With the cache, projection per step is O(1) for the new token, and attention is a single O(N) pass over the stored keys and values. The price you pay is HBM memory, since the cache grows linearly with sequence length.
Quick reference
TL;DR
The KV cache stores past tokens' K and V projections per layer and head, so each decode step computes K, V only for the new token instead of recomputing the whole prefix.
Imagine writing a story where every new sentence has to stay consistent with everything you wrote before. Without notes, you would reread the entire story before writing each sentence. The KV cache is a notebook: for each sentence you finish, you jot a short label (the key) and a copy of its meaning (the value). To write the next sentence, you just scan your labels and pull the matching meanings, instead of rereading the whole story. You only add one new note per sentence. The fresh thought you have right now (the query) gets used immediately and never stored, because no future sentence ever looks back at it. So the notebook holds only labels and meanings, never your in the moment questions.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short prompts and very short generations on a small model The bookkeeping to manage a cache can exceed the recomputation it avoids when total tokens are tiny.
Tight HBM budgets at very long context The cache can dwarf the weights, so you must quantise it to fp8 or int8 or accept a far smaller batch.
Training rather than inference Training processes the whole sequence in parallel each step, so there is no autoregressive prefix to reuse across steps.
Forking generation many ways without paged sharing Beam search or branched sampling needs a cache copy per branch, multiplying memory unless prefixes are shared.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name what is cached (K and V, per layer per head per position) and what is not (Q, weights), then the O(N^2) to O(N) per-step argument, then the HBM cost and the levers that manage it.
Real-world examples
Real products, models, and research that use this idea.
1vLLM (UC Berkeley) implements paged attention over the KV cache and is the basis of most open-source LLM serving in 2026.
2DeepSeek V4 ships Multi-head Latent Attention, which compresses cached K and V to a low-rank latent to shrink the per-request cache.
3Llama 4 and Mistral Large 3 use Grouped-Query Attention to cut the number of cached KV heads versus full multi-head attention.
4TensorRT-LLM (NVIDIA) provides paged KV cache and fp8 KV as the production reference on H100 and B200 hardware.
5Anthropic and OpenAI expose prompt caching, which keeps the KV state of a repeated prompt prefix warm to cut input cost on reuse.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is Q not cached when K and V both are?
Sign in or go Pro to see the approach for this follow-up.
QWhat dominates the KV cache memory footprint at scale?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the cache make decode memory bound rather than compute bound?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying the cache stores Q or attention weights. Past queries are never re-read and weights are recomputed each step; only the K and V of prior tokens persist.