Why is KV cache memory the bottleneck for long context LLM serving?
Derive the KV cache memory formula. Why does it (rather than model weights) become the dominant memory cost at long context, and how does GQA mitigate this?
Weights are flat; KV cache scales with context times batch and surpasses weights fast. GQA cuts kv_heads for integer savings, Llama-3 70B drops 64 heads to 8 KV heads for 8x.
Picture the model as a fixed encyclopedia, same shelf space whether you ask one question or a thousand. The model's working notes (the KV cache) are different: every word in every active conversation adds a new note, and the stack of notes grows with both how long each chat gets and how many chats are happening at once. Past a certain context length, the notes take more room than the encyclopedia. GQA shrinks each note by letting groups of readers share one set of K and V slots, Llama-3 70B uses 8 shared sets across 64 readers, so the note pile shrinks 8x.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Derive the formula, contrast with the fixed weight memory, walk a concrete crossover example, explain GQA's mechanism and savings ratio, place it in the MHA→MQA→GQA→MLA arc, and tie to production levers (paged allocation, KV quantization, prefix sharing).
def cache_vs_weights(num_params_billion, num_layers, num_kv_heads, d_head,
bytes_per_value, seq_len, batch_size):
weights_gb = num_params_billion * bytes_per_value
per_req_cache_gb = (2 * num_layers * num_kv_heads * d_head
* bytes_per_value * seq_len) / 1e9
total_cache_gb = per_req_cache_gb * batch_size
crossover = total_cache_gb / weights_gb
return dict(weights_gb=weights_gb,
per_req_cache_gb=round(per_req_cache_gb, 2),
total_cache_gb=round(total_cache_gb, 2),
cache_to_weight_ratio=round(crossover, 2))
# Llama-3 70B, fp16, 32k context, batch 16, GQA-8
print(cache_vs_weights(70, 80, 8, 128, 2, 32768, 16))
# weights ~140 GB, total cache ~172 GB — cache exceeds weights
# Same model, but textbook MHA (64 KV heads), batch 16, 32k
print(cache_vs_weights(70, 80, 64, 128, 2, 32768, 16))
# total cache ~1376 GB — completely infeasible| Variant | num_kv_heads (Llama-2 70B shape) | Cache savings vs MHA | Quality impact |
|---|---|---|---|
| MHA | 64 (= num_query_heads) | 1x baseline | baseline |
| MQA | 1 | 64x | Noticeable on some tasks |
| GQA (G=8) | 8 | 8x | Near-MHA, production standard |
| MLA | n/a (low rank latent ~ 512 dim) | ≈10-20x | Comparable to MHA per DeepSeek-V2 |
Real products, models, and research that use this idea.
- vLLM benchmarks in 2026 routinely show KV cache exceeding model weight memory in long context serving, which is exactly what PagedAttention was designed to absorb.
- Claude Opus 4.7 long context serving relies heavily on cache management; at 200k context, per-request cache is dozens of GB even with aggressive grouping.
- DeepSeek V4 uses MLA and reports serving cost an order of magnitude below comparable MHA models specifically because of the latent cache footprint.
- Llama 4 Maverick ships with GQA so it fits production serving at long context where a textbook MHA equivalent would not.
- Mistral Large 3 with GQA fits 32k-plus context on a single 8xH200 node; the same arithmetic with MHA would overflow even at modest batch sizes.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you doubled context length but halved batch size, how does total system memory change?
QWhy doesn't everyone use MLA if it's the smallest cache?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Believing model weights are the binding memory constraint at long context, they're fixed; KV cache is what scales, and it surpasses weights quickly at moderate context × batch.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.