During decode, why is only Q computed for the new token while full K and V come from cache?
Each decode step makes one new Q row that reads all past K, V once and is never reread. K and V are read by every future query, so only K and V get cached.
Picture a long meeting where every minute someone new walks in, asks one question, listens to everything everyone has ever said, writes down one sentence, then leaves the room forever. The questions are throwaway: nobody else needs to look at the question someone asked five minutes ago. But the things everyone else said? Every future newcomer will replay those. So you bother to remember (cache) the said things, and you happily redo the throwaway question each time. That is exactly what a transformer does during decode with Q versus K and V.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Understanding why the KV cache exists, and why it is K and V that get cached rather than Q, is one of those questions that separates people who have implemented a transformer decoder from people who have only read about one. The answer is not 'Q is cheaper'. The answer is about the lifetime of each tensor row in an autoregressive loop.
This deep dive walks one decode step in detail, contrasts it with prefill, derives the KV-cache memory formula that drives modern architecture choices like GQA and MLA, and explains why every production inference engine treats the cache as its dominant memory-budget line item in 2026.
The payoff is a clean mental model that explains GQA, MQA, MLA, paged attention, prefix caching, and KV-cache offload as variations on the same theme: only cache tensors that get reread.
Decode step, traced explicitly
Suppose the model has already emitted tokens at positions 0 through t-1 and is about to emit position t. The input at this step is the single embedding for the just-emitted token at position t-1 (or the BOS plus prompt during prefill).
What happens inside one attention layer
- The fresh embedding x_t goes through
W_Q,W_K,W_V, producing one row each: q_t, k_t, v_t (shape(1, d_head)per head). - k_t and v_t are APPENDED to the cache: K becomes
(t+1, d_head), V becomes(t+1, d_head). - The attention score is
q_t @ K^Twhich is shape(1, t+1), divided by sqrt(d_k) and softmaxed. - The output context vector is
softmax(...) @ V, shape(1, d_head). - That context vector becomes the input to the FFN, then to the next block.
What is read versus written
- Written this step: k_t, v_t (one row each, appended to cache).
- Read this step: all rows of K and V (positions 0..t).
- Read once and never again: q_t.
That last bullet is the entire intuition. q_t lives for exactly one matmul, then is dropped on the floor. K and V live for the rest of the conversation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM's PagedAttention manages KV-cache pages explicitly because KV is the dominant memory budget at long context for Llama 4 Maverick serving.
- DeepSeek V4's MLA (Multi-head Latent Attention) compresses the KV cache further by sharing a low-rank latent across heads.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through what changes about KV-cache behavior during prefill versus decode.
Prefill processes the whole prompt in one batched matmul: all Q, K, V are computed in parallel and K, V are written into the cache. Decode then reads K, V row by row while only generating one new q_t per step. Prefill is compute-bound, decode is memory bandwidth bound, which drives different kernel choices and continuous batching tricks.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming Q is cheaper than K or V or that caching past queries would save work. The asymmetry is about reuse: Q is read once, K and V are reread every step.
60 second bullets to scan on the way to the call.
What gets produced fresh at each decode step
Lifetime of a Q row versus a K or V row
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.