What is the KV cache in transformer inference?
The KV cache stores the keys and values of every prior token so each new decode step only needs one new K and V, turning O(n^2) attention into O(n) per step.
Imagine writing a long essay where every sentence has to reference all previous sentences. Without notes, you would reread the whole essay before writing each new sentence. The KV cache is your scratchpad, you jot down a short note for every sentence you have already written (the key) and a copy of its content (the value). When writing the next sentence, you just glance at the notes instead of rereading. Each new sentence only adds one row to the scratchpad. Reading gets faster as the essay grows, because the work per new word stays small even though the essay gets long.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The KV cache is the single most important data structure in modern LLM serving. It is also the single most common interview blind spot, because candidates who have only used the API never see it, and candidates who learned attention from the original Transformer paper often miss the inference-time specifics.
A serving stack is essentially a system organised around the cache. The model weights are static; the cache grows and shrinks per request, fragments under load, dominates HBM at long context, and sets the throughput ceiling on every GPU. Every optimisation that matters in 2026 (paged attention, fp8 KV, speculative decoding, prefix caching, GQA, MLA) is in some sense a response to a constraint that the cache imposes on the serving system.
This deep dive walks through what the cache is, why it exists, what it costs, what architectural tricks shrink it, and what serving-side techniques manage it. By the end you should be able to do cache math on a napkin and explain why a 70B model with 128k context can be more memory-pressured by the cache than by its own weights.
The recomputation problem the cache solves
In autoregressive decoding the model emits one token at a time. To emit token T+1, the attention layer at each level needs the keys and values of every previous token from 1 to T. A naive implementation projects all T previous tokens into K and V at every step.
That's O(T) projection work per step, plus O(T) attention work, so generating N tokens from a length-T prompt is O((T+N)^2) total. At T=8000 and N=500 this is roughly 70 million extra projection operations per layer, multiplied by 32 to 80 layers. Throughput collapses.
The key insight: K and V for past tokens are deterministic functions of those tokens' hidden states, which never change once produced. Cache them once and re-read. Work per new token drops to O(T+t) attention plus O(1) projection.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM (UC Berkeley) introduced paged attention as the default KV cache manager and is the basis of most open-source serving in 2026.
- SGLang serves Llama 4 and Qwen 3 with native KV cache prefix sharing across batched requests for common system prompts.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is Q not cached if K and V are?
Q for past tokens is never re-read after that token's own attention layer completes. Each new decode step generates a fresh Q for the new token; old Qs play no role in the new step's attention. Cache only what gets re-read.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying the KV cache stores Q (queries) or attention weights. Past Qs are never reused, and weights are recomputed every step; only K and V from prior tokens persist.
60 second bullets to scan on the way to the call.
What persists in the cache versus what is recomputed each step
Whether Q is cached, and why or why not
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.