What exactly does the KV cache store and what computational redundancy does it eliminate?

In a transformer, each token is projected into three vectors: a query Q, a key K, and a value V. The attention output for a token is a weighted sum of every value, where the weights come from that token's query dotted against every key. When the model generates text, it emits one token at a time. To produce the next token, its attention layers need the keys and values of every token that came before. A naive decoder would re-project all of those past tokens into K and V on every single step, which is wasteful because those projections never change. The insight is that a past token's K and V are fixed functions of that token's hidden state, and that hidden state is frozen once the token is generated. So you compute each token's K and V once and keep them. The **KV cache** is that storage, organised by layer, by head, and by token position. On each decode step you compute K and V only for the one new token, append them, and read the whole cache to do attention. Q for past tokens is never kept, because no later step ever looks at it again.

**The KV cache** stores the key and value projections of every token already processed, indexed by layer, head, and position. It exists so autoregressive decoding can attend to the full history without re-projecting the prefix each step. Mechanism. At each decode step the model computes Q, K, and V for the single new token only. It appends that token's K and V to the cache. The new query then dot-products against all cached keys, softmaxes, and forms a weighted sum over all cached values. The asymmetry is the crux. K and V are cached because every future token will attend to them. Q is recomputed fresh and never stored, because a past token's query is consumed once during its own step and never read again. This is what collapses the cost. Without the cache, regenerating K and V for the whole prefix at every step is O(N) work per step, so O(N^2) over a full completion. With the cache, projection per step is O(1) for the new token, and attention is a single O(N) pass over the stored keys and values. The price you pay is HBM memory, since the cache grows linearly with sequence length.