What is the KV cache in transformer inference?
The KV cache stores the keys and values of every prior token so each new decode step only needs one new K and V, turning O(n^2) attention into O(n) per step.
Imagine writing a long essay where every sentence has to reference all previous sentences. Without notes, you would reread the whole essay before writing each new sentence. The KV cache is your scratchpad: you jot down a short note for every sentence you have already written (the key) and a copy of its content (the value). When writing the next sentence, you glance at the notes instead of rereading. Each new sentence only adds one row to the scratchpad. Writing stays quick as the essay grows, because the work per new word stays small even though the essay gets long.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The KV cache is the most important data structure in LLM serving. It is easy to miss when working only through an API, and the original Transformer paper does not cover these inference-time details.
A serving stack is organised around the cache. The model weights are static. The cache grows and shrinks per request, fragments under load, takes most of HBM at long context, and limits throughput on every GPU. Every major serving optimisation (paged attention, fp8 KV, speculative decoding, prefix caching, GQA, MLA) responds in some way to a limit that the cache puts on the serving system.
At 128k context, a few Llama 3 70B requests need more cache memory than the 140 GB of bf16 weights.
The recomputation problem the cache solves
In autoregressive decoding the model emits one token at a time. To emit token T+1, the attention layer at each level needs the keys and values of every previous token from 1 to T. A naive implementation projects all T previous tokens into K and V at every step.
That's O(T) projection work per step on top of the attention work, and it adds up over every step of the generation. For an 8,000-token prompt and 500 generated tokens, the model would re-project about 4.1 million past token positions per layer (500 steps of about 8,250 tokens each), multiplied by 32 to 80 layers. Throughput drops sharply.
K and V for past tokens are deterministic functions of those tokens' hidden states, which never change once produced. So they can be computed once, cached and re-read. Work per new token drops to O(T) attention plus O(1) projection.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM (UC Berkeley) introduced PagedAttention (Kwon et al., SOSP 2023), manages its KV cache in 16-token blocks by default, and is the basis of most open-source serving.
- SGLang serves Llama 4 and Qwen 3 with RadixAttention, which keeps the KV cache of finished requests in a radix tree. A new request reuses the longest cached prefix, such as a shared system prompt.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is Q not cached if K and V are?
Q for past tokens is never re-read after that token's own attention layer completes. Each new decode step generates a fresh Q for the new token. Old Qs play no role in the new step's attention. Cache only what gets re-read.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying the KV cache stores Q (queries) or attention weights. Past Qs are never reused, and weights are recomputed every step. Only K and V from prior tokens persist.
60 second bullets to scan on the way to the call.
What persists in the cache versus what is recomputed each step
Whether Q is cached, and why or why not
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.