During decode, why is only Q computed for the new token while full K and V come from cache?

**Decode** in an autoregressive transformer emits one new token at a time. At step t the model needs exactly one thing: the next token's distribution. To compute it, the attention layer needs one new query (the projection of the freshly arrived token) and the full set of keys and values from every earlier position. The key observation is asymmetric reuse. A query row at position t is used exactly once: in step t, to weight K and V from positions 0..t. Once step t finishes, no future step ever reads q_t again. Future steps each produce their own new query. Keys and values are the opposite. The K and V rows for position 3 get reread at step 4, step 5, step 6, and every later step. Without a cache, you would reproject all earlier tokens through W_K and W_V on every decode step, which is O(t) repeated work per layer per step. So the engineering decision is: spend memory caching the tensors that get reread (K and V) and accept the cheap recomputation of the tensor that is read once (Q). That asymmetry is the entire reason 'KV cache' is named the way it is.

Decode is autoregressive: one new token per step. At step t exactly one new query row is generated (the projection of the just emitted token), and that query attends across all past keys and values to produce one context vector. The asymmetry is reuse. Each q_t is consumed once, on its own step, then discarded. K and V at position j get reread by every future query q_{j+1}, q_{j+2}, q_{j+3} and so on. Caching only pays for tensors that are read more than once. If you cached Q you would be storing rows nothing downstream looks at. If you did not cache K and V, every decode step would have to reproject the entire prompt and all previously generated tokens through `W_K` and `W_V`, which is O(t) extra projection work per layer per step. Over a 4k-token generation that turns a linear cost into a quadratic one. This is why long-context inference engines (vLLM, TensorRT-LLM, SGLang) treat the KV cache as the dominant memory budget item, and why MQA, GQA, and MLA, schemes that shrink K and V, are 2024-2026 deployment standards.