Llama-2 70B has 80 layers, 64 attention heads, d_head = 128. Assume MHA (not GQA, use original Vaswani style). Context length = 4096. Compute the KV cache memory in FP16 (2 bytes per value). Show the formula.
Llama-2 70B MHA at 4k context in fp16 needs ~10 GB of KV cache per request; GQA-8 collapses that to ~1.3 GB by cutting kv_heads from 64 to 8.
Think of every word the model reads as leaving two sticky notes on each shelf of a tall library, one note labeled K, one labeled V. For a Llama sized library with 80 shelves and 64 reading stations per shelf, a 4096-word document fills enough notes to fill a small filing cabinet (about 10 GB of GPU memory). Pile up a few visitors at once and the cabinet overflows. Newer designs let every 8 reading stations share one set of notes, shrinking the cabinet 8x without losing what the readers can do.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
If you only memorize one piece of arithmetic in LLM inference, make it the KV cache memory formula. It governs how many concurrent requests a GPU can hold, how far you can push context length, and why every flagship model since 2023 has shipped GQA, MQA, or MLA. The arithmetic is brutally simple, six factors multiplied together, but the consequences cascade across architectural design, kernel implementation, and serving economics.
This deep dive walks the six factors, plugs in concrete numbers for Llama-2 70B under both the textbook MHA assumption and the GQA-8 reality, traces the scaling behavior across context and batch, then connects the formula to every downstream optimization the field has invented since 2022: paged allocation, KV quantization, prefix sharing, MLA's latent reshape, and the FlashAttention family of kernels.
The goal is that after reading you can take any config.json, ballpark per-request KV memory at arbitrary context, predict batch capacity on a given GPU class, and explain why a particular architectural choice was made. That diagnostic ability is what separates engineers who can size a serving fleet from those who guess.
The six factors and what each one controls
Drop any one factor and the estimate is off by an integer ratio that's almost always wrong by an order of magnitude.
The leading 2 counts K and V as separate tensors of identical shape. Some learners try to fold it into other factors; don't. Keep it explicit so the formula stays mechanical.
L is transformer depth. Every block stores its own K/V tensors because each attention operation needs its own representations. This is the most commonly forgotten factor in interview answers. Drop it on a 70B model and you're off by 80x.
H_kv is the number of distinct K/V projections. Under MHA it equals query head count. Under MQA it collapses to 1. Under GQA with group size G, it becomes num_query_heads / G. Under MLA (DeepSeek-V2, 2024) the formula breaks entirely.
d_h is per-head dimension, typically 64, 96, or 128. Set during architecture design and rarely changed.
T is the cached sequence length, growing one token per decode step.
b is bytes per value: 4 for fp32, 2 for fp16/bf16, 1 for int8 or fp8 (E4M3 on Hopper+), roughly 0.5 for int4 packed.
The factors compound. Halving H_kv (GQA) and halving b (int8 KV) together gives 4x reduction, not 2x. Every modern optimization stacks orthogonally on this formula.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
def kv_cache_bytes(num_layers, num_kv_heads, d_head, bytes_per_value, seq_len):
"""Per-request KV cache memory in bytes."""
return 2 * num_layers * num_kv_heads * d_head * bytes_per_value * seq_len
# Llama-2 70B as MHA (textbook), 4k context, fp16
print(kv_cache_bytes(80, 64, 128, 2, 4096) / 1e9, 'GB') # ~10.74 GB
# Llama-2 70B as it actually ships (GQA, 8 KV heads), 4k context, fp16
print(kv_cache_bytes(80, 8, 128, 2, 4096) / 1e9, 'GB') # ~1.34 GB
# Same model at 32k context, GQA, fp16
print(kv_cache_bytes(80, 8, 128, 2, 32768) / 1e9, 'GB') # ~10.7 GB| Setting | Cache per token | Cache @ 4k context | Cache @ 32k context |
|---|---|---|---|
| Llama-2 70B MHA (textbook), fp16 | 2.5 MB | 10.74 GB | 85.9 GB |
| Llama-2 70B real (GQA-8), fp16 | 320 KB | 1.34 GB | 10.74 GB |
| Llama-2 70B GQA-8, int8 KV | 160 KB | 0.67 GB | 5.37 GB |
| Llama-2 70B MQA (hypothetical), fp16 | 40 KB | 0.17 GB | 1.34 GB |
Real products, models, and research that use this idea.
- Llama 4 Maverick uses GQA with 8 KV heads on 64 query heads; per-request cache at 32k context drops 8x vs textbook MHA.
- Mistral Large 3 with 8 KV heads at 64k context in fp16 produces a per-request cache larger than the model weights for batch sizes above 4.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf your GPU has 80 GB and the model occupies 70 GB after sharding, how many simultaneous requests can you serve at 4k context with Llama-2 70B GQA?
Remaining memory ≈ 10 GB. Per-request cache ≈ 1.34 GB. Naive division: ~7 requests. In practice, paged attention can squeeze more by avoiding over reservation; also prefix sharing helps if requests share a system prompt.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting the factor of 2 for K + V, using num_query_heads instead of num_kv_heads (matters under GQA/MQA), or using fp32 bytes (4) instead of fp16/bf16 (2).
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
- Shazeer 2019 — Fast Transformer Decoding: One Write-Head is All You Need (MQA)
- Ainslie et al. 2023 — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-AI 2024 — DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (introduces MLA)
Same topic, related formats. Practice these next.