A memory of the keys and values computed for every prior token. During autoregressive decoding, the next token only needs to attend to those cached K/V, saving O(n²) recomputation per step.
**Transformer attention** projects each input token into three vectors: query Q, key K, and value V. The attention output for a token is a weighted sum of all token values, where the weights come from the dot product between this token's Q and every other token's K.
In autoregressive generation, the model emits one token at a time. Naively, every decode step would recompute K and V for every previous token, which is O(n^2) work at step n. That balloons fast as the generation gets long.
The key observation: for any past token, K and V do not change. They are deterministic functions of that token's hidden state, and that hidden state is fixed once the token is generated. So we can compute K and V once per token, store them, and reuse them.
The **KV cache** is that storage. At each decode step, we compute K and V for the single new token, append them to the cache, and the attention layer reads the entire cache to compute attention scores. Work per step drops from O(n) to O(1) for projection, with attention itself becoming O(n) instead of O(n^2).
**The KV cache** stores the keys and values of every previously processed token, indexed by layer and head, so autoregressive decoding can attend to history without recomputing past projections.
Mechanism: at each decode step, the model computes Q, K, V only for the single new token. It appends the new K and V to the per-layer, per-head cache. The attention layer then computes the new token's Q against ALL cached keys. It applies a softmax-weighted sum over ALL cached values.
The cache size formula is `2 * L * H_kv * d_h * T * b` bytes. Here L is layers. `H_kv` is KV heads. `d_h` is head dim. T is sequence length. b is bytes per element. `H_kv` is 1 for MQA. It is fewer than `n_heads` for GQA. For a 70B model at 8k context in fp16, this is roughly 5 to 10 GB per request, often larger than the model itself in memory pressure.
The cache is what makes long-context decoding tractable; managing it is the central concern of modern serving stacks (paged attention, vLLM, GQA, MLA).
Quick reference
TL;DR
The KV cache stores the keys and values of every prior token so each new decode step only needs one new K and V, turning O(n^2) attention into O(n) per step.
Imagine writing a long essay where every sentence has to reference all previous sentences. Without notes, you would reread the whole essay before writing each new sentence. The KV cache is your scratchpad, you jot down a short note for every sentence you have already written (the key) and a copy of its content (the value). When writing the next sentence, you just glance at the notes instead of rereading. Each new sentence only adds one row to the scratchpad. Reading gets faster as the essay grows, because the work per new word stays small even though the essay gets long.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very small prompts and very short generations on small models Cache management overhead exceeds the recomputation it avoids. Recompute is fine under a few hundred tokens total.
Memory-constrained edge deployment with tight HBM budget At 128k context the cache dwarfs the model. Strategies like quantising the cache to int8 or fp8 become mandatory.
Use cases that fork generation many times (beam search, branched sampling) Each branch needs its own cache copy unless paged attention lets you share prefixes; naive caching multiplies memory.
Training rather than inference Training computes Q, K, V for the entire batch in parallel each step; there is no autoregressive dependency to cache across.
Quick scan · 2–4 min
Everything important, quickly.
3 min: prefill versus decode + what's cached (K, V) + cache size formula with num_layers + GQA and MLA + paged attention serving.
Real-world examples
Real products, models, and research that use this idea.
1vLLM (UC Berkeley) introduced paged attention as the default KV cache manager and is the basis of most open-source serving in 2026.
2SGLang serves Llama 4 and Qwen 3 with native KV cache prefix sharing across batched requests for common system prompts.
3DeepSeek V4 ships Multi-head Latent Attention specifically to shrink the per-request cache by 10x compared to MHA.
4Anthropic's Claude Opus 4.7 serving stack uses cache-aware routing to colocate requests sharing long system prompts on the same replica.
5TensorRT-LLM (NVIDIA) implements paged KV cache and chunked prefill as the production reference on H100 and B200 hardware.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is Q not cached if K and V are?
Sign in or go Pro to see the approach for this follow-up.
QHow does GQA change the cache memory math compared to MHA?
Sign in or go Pro to see the approach for this follow-up.
QWhat does paged attention solve that naive KV caching does not?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying the KV cache stores Q (queries) or attention weights. Past Qs are never reused, and weights are recomputed every step; only K and V from prior tokens persist.