Predict the KV cache size in GB for given model dimensions
Given a transformer with the following dimensions, predict the KV cache size in GB (to one decimal place) for a single request at full context. Use the standard formula `bytes = 2 * n_layers * n_kv_heads * head_dim * seq_len * dtype_bytes`. Inputs: - n_layers = 80 - n_kv_heads = 8 (this model uses GQA with group size 8) - head_dim = 128 - seq_len = 131072 (128k tokens) - dtype = FP16 (2 bytes per element) - batch = 1 Report the result in GB (1 GB = 1024^3 bytes). Round to one decimal place.
Multiply 2 (K and V) by layers, KV heads, head dim, sequence length, and bytes per element. The result is about 20.0 GB at 128k context.
Picture a hotel where every guest who has ever checked in keeps their room forever, and you have to walk past all those rooms each time someone new arrives. The KV cache is that hallway of kept rooms. Its size is just a chain of multiplications: how many floors (layers), how many rooms per floor (KV heads), how big each room is (head dimension), how many guests so far (sequence length), and how many bytes each room costs (the number format). Multiply them all, then double it because every guest needs two rooms, one for keys and one for values. Do that for a long-context request and you discover the hallway can grow larger than the whole hotel building, which is the model weights.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: write the formula, plug in the six numbers, multiply carefully, convert with 1024 cubed, then explain the 8x GQA ratio versus the MHA baseline.
Real products, models, and research that use this idea.
- Llama 3.1 70B uses GQA with 8 KV heads, the exact configuration in this problem, so this 20 GB figure mirrors its real per-request long-context footprint.
- vLLM sizes its paged KV blocks using precisely this byte formula to decide how many concurrent requests fit in the remaining HBM after weights.
- DeepSeek V4 reports per-request cache savings against an MHA baseline by computing this same product with and without the head reduction.
- NVIDIA TensorRT-LLM exposes a KV cache memory calculator that implements this formula to plan H100 and B200 deployments.
- Mistral Large 3 ships GQA so its 128k context cache stays in the tens of GB rather than the hundreds an MHA design would demand.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat would this same model's cache cost if it used full MHA with 64 KV heads instead of GQA?
QHow does the answer change at fp8 KV cache instead of fp16?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Dropping the layer count, or forgetting the leading factor of 2 for K and V. Either error throws the answer off by an order of magnitude or by half.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.