What exactly does the KV cache store and what computational redundancy does it eliminate?
Explain what the KV cache stores during autoregressive decoding and what work it lets you skip on each new token step. Be specific about which tensors are cached and which are recomputed.
The KV cache stores past tokens' K and V projections per layer and head, so each decode step computes K, V only for the new token instead of recomputing the whole prefix.
Imagine writing a story where every new sentence has to stay consistent with everything you wrote before. Without notes, you would reread the entire story before writing each sentence. The KV cache is a notebook: for each sentence you finish, you jot a short label (the key) and a copy of its meaning (the value). To write the next sentence, you just scan your labels and pull the matching meanings, instead of rereading the whole story. You only add one new note per sentence. The fresh thought you have right now (the query) gets used immediately and never stored, because no future sentence ever looks back at it. So the notebook holds only labels and meanings, never your in the moment questions.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The KV cache is the data structure that makes autoregressive decoding affordable, and this question targets the two facts interviewers care about most: exactly what it stores, and exactly what redundant work it removes. Many candidates can recite that a cache exists but stumble when asked to name the tensors and justify why one of the three attention projections is deliberately thrown away.
The heart of the answer is an asymmetry in the attention dataflow. Keys and values for a token are read by every token that comes after it, so they are worth keeping. The query for a token is used once, during that token's own step, and never again, so keeping it would waste memory for no benefit.
This deep dive walks through the recomputation that the cache eliminates, exactly which tensors are stored and which are rebuilt, the per-step and full-completion complexity, the prefill versus decode split, and the memory cost that the cache trades in return. By the end you should be able to defend each design choice rather than just describe the artifact.
The redundant work the cache eliminates
In autoregressive decoding the model emits one token per step. To produce token T+1, every attention layer needs the keys and values of all tokens from 1 to T. A decoder with no cache would re-project all T prior tokens into K and V on every step, because the next forward pass starts from scratch with the whole sequence as input.
That is O(T) projection work per step on top of O(T) attention work. Generating N tokens from a length-T prompt becomes roughly O((T+N)^2) total. At a few thousand tokens of context, across 32 to 80 layers, this recomputation alone is enough to collapse throughput. The painful part is that it scales with completion length, so the longer the model talks, the slower each subsequent token gets.
The redundancy is pure waste because a past token's K and V are deterministic functions of its hidden state, and that hidden state is frozen the moment the token is generated. The hidden state of token 5 never changes after step 5, so its key and value never change either. Computing them again produces bit for bit the same tensors.
Cache them once, read them forever, and per-step projection drops to a single new token. The cache does not change the math of attention at all; it changes only how many times you pay to materialise the keys and values that go into it. That is the whole trick, and it is why the cache is correctness-neutral: outputs are identical with or without it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM (UC Berkeley) implements paged attention over the KV cache and is the basis of most open-source LLM serving in 2026.
- DeepSeek V4 ships Multi-head Latent Attention, which compresses cached K and V to a low-rank latent to shrink the per-request cache.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is Q not cached when K and V both are?
Trace the dataflow. A token's query scores against keys only during that token's own step, then is discarded. Future steps generate their own fresh queries. Cache only what later steps re-read, which is past K and V.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying the cache stores Q or attention weights. Past queries are never re-read and weights are recomputed each step; only the K and V of prior tokens persist.
60 second bullets to scan on the way to the call.
Which tensors persist in the cache and which are recomputed each step
Why Q is not cached but K and V are
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.