What is the KV cache in transformer inference?

**Transformer attention** projects each input token into three vectors: query Q, key K, and value V. The attention output for a token is a weighted sum of all token values, where the weights come from the dot product between this token's Q and every other token's K. In autoregressive generation, the model emits one token at a time. Naively, every decode step would recompute K and V for every previous token, which is O(n^2) work at step n. That balloons fast as the generation gets long. The key observation: for any past token, K and V do not change. They are deterministic functions of that token's hidden state, and that hidden state is fixed once the token is generated. So we can compute K and V once per token, store them, and reuse them. The **KV cache** is that storage. At each decode step, we compute K and V for the single new token, append them to the cache, and the attention layer reads the entire cache to compute attention scores. Work per step drops from O(n) to O(1) for projection, with attention itself becoming O(n) instead of O(n^2).

**The KV cache** stores the keys and values of every previously processed token, indexed by layer and head, so autoregressive decoding can attend to history without recomputing past projections. Mechanism: at each decode step, the model computes Q, K, V only for the single new token. It appends the new K and V to the per-layer, per-head cache. The attention layer then computes the new token's Q against ALL cached keys. It applies a softmax-weighted sum over ALL cached values. The cache size formula is `2 * L * H_kv * d_h * T * b` bytes. Here L is layers. `H_kv` is KV heads. `d_h` is head dim. T is sequence length. b is bytes per element. `H_kv` is 1 for MQA. It is fewer than `n_heads` for GQA. For a 70B model at 8k context in fp16, this is roughly 5 to 10 GB per request, often larger than the model itself in memory pressure. The cache is what makes long-context decoding tractable; managing it is the central concern of modern serving stacks (paged attention, vLLM, GQA, MLA).