Only K and V projections of past tokens, per layer and per KV head, live in the KV cache. Queries are used once and thrown away; FFN activations and raw embeddings are not cached.
Imagine the model is taking turns adding words to a story. At every new word, it asks: which of the earlier words should I pay attention to? To answer that question, the model needs two things about each earlier word: a label saying what kind of information that word offers (the key) and the actual information itself (the value). It does not need to remember the question each earlier word asked, because that question was for that word's own turn. So the inference server keeps a library of labels and values for every past word, layer by layer, and discards the questions. That library is the KV cache. It does not store FFN intermediates or embeddings because those are easy to recompute, and they are not what the next attention step needs.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The KV cache is the most important piece of state in modern LLM inference. At long context and high batch, it eats more GPU memory than the model weights, and every architecture choice from GQA to MLA exists to shrink it. So a clean answer to 'what is actually in there?' is foundational.
The short answer: K and V projections of every past token, per layer and per KV head. Nothing else. This walkthrough explains why the other obvious candidates (Q, FFN activations, raw embeddings) are not cached, traces what really lives in cache through one decode step, and ends with how modern serving stacks lay this out physically with paged attention.
Walk through one decode step
At decoding step t, the model has already generated tokens 0 through t-1 and is producing token t. Here is what happens at the attention sublayer of one layer:
- The new token's hidden state x_t enters the sublayer.
- The layer computes its Q, K, V projections from x_t alone:
q_t = x_t W_Q,k_t = x_t W_K,v_t = x_t W_V. These are single-token-sized tensors. - The new k_t and v_t are appended to the cache, which already holds k_0..k_{t-1} and v_0..v_{t-1} from previous steps.
- Attention computes
softmax(q_t K^T / sqrt(d_head)) V, where K and V are the full cached tensors including the newly appended k_t and v_t. - The attention output is added to the residual stream and the sublayer finishes.
At step t+1, the same thing happens again: a new q_{t+1} is computed from x_{t+1} and attended against the cache, which now has one more entry. Notice that q_t from step t is never used again; it lived for one matmul. K and V from every past step are read on every subsequent step.
That asymmetry is the whole reason the cache stores K and V but not Q.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and TensorRT-LLM serve modern LLMs with paged KV caches; their cache eviction and sharing policies operate on K and V blocks only.
- Llama 3 70B at 8k context uses roughly 2.5 GB of KV cache per sequence in fp16; weights are 140 GB, so cache becomes dominant only at high batch.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do GQA, MQA, and MLA all reduce K and V dimensions but leave Q untouched?
Q is not cached, so shrinking Q would only save compute (negligibly) and would hurt expressivity. K and V dominate decode-time memory and bandwidth, so they are the right lever. GQA shares K and V across query groups; MQA collapses to one head; MLA stores a compressed latent.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Caching Q (which is consumed once per token) or FFN activations (which are position-wise and never revisited). The KV cache is specifically K and V because those are the tensors a new token's query reads from later steps.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.