Compute the KV cache memory for a single request at 4096 context on Llama-2 70B (MHA) in FP16.

Predict output·Hard·4.0 · 0·~2 min·Relevant atAnthropic NVIDIA

Attempt it

Llama-2 70B has 80 layers, 64 attention heads, d_head = 128. Assume MHA (not GQA, use original Vaswani style). Context length = 4096. Compute the KV cache memory in FP16 (2 bytes per value). Show the formula.

Quick reference

TL;DR

Llama-2 70B MHA at 4k context in fp16 needs ~10 GB of KV cache per request; GQA-8 collapses that to ~1.3 GB by cutting kv_heads from 64 to 8.

Memory aid

Easy to grasp

Think of every word the model reads as leaving two sticky notes on each shelf of a tall library, one note labeled K, one labeled V. For a Llama sized library with 80 shelves and 64 reading stations per shelf, a 4096-word document fills enough notes to fill a small filing cabinet (about 10 GB of GPU memory). Pile up a few visitors at once and the cabinet overflows. Newer designs let every 8 reading stations share one set of notes, shrinking the cabinet 8x without losing what the readers can do.

Key concepts

KV cache KV cache memory formula Grouped Query Attention (GQA)

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

State the formula, plug in numbers, then layer in GQA savings (8x), int8/fp8 quantization (2x), MLA's latent compression, and the implications for GPU memory budgeting at production batch sizes and context lengths.

Code example

python

def kv_cache_bytes(num_layers, num_kv_heads, d_head, bytes_per_value, seq_len):
    """Per-request KV cache memory in bytes."""
    return 2 * num_layers * num_kv_heads * d_head * bytes_per_value * seq_len

# Llama-2 70B as MHA (textbook), 4k context, fp16
print(kv_cache_bytes(80, 64, 128, 2, 4096) / 1e9, 'GB')   # ~10.74 GB

# Llama-2 70B as it actually ships (GQA, 8 KV heads), 4k context, fp16
print(kv_cache_bytes(80, 8, 128, 2, 4096) / 1e9, 'GB')    # ~1.34 GB

# Same model at 32k context, GQA, fp16
print(kv_cache_bytes(80, 8, 128, 2, 32768) / 1e9, 'GB')   # ~10.7 GB

Side-by-side

Setting	Cache per token	Cache @ 4k context	Cache @ 32k context
Llama-2 70B MHA (textbook), fp16	2.5 MB	10.74 GB	85.9 GB
Llama-2 70B real (GQA-8), fp16	320 KB	1.34 GB	10.74 GB
Llama-2 70B GQA-8, int8 KV	160 KB	0.67 GB	5.37 GB
Llama-2 70B MQA (hypothetical), fp16	40 KB	0.17 GB	1.34 GB

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick uses GQA with 8 KV heads on 64 query heads; per-request cache at 32k context drops 8x vs textbook MHA.
Mistral Large 3 with 8 KV heads at 64k context in fp16 produces a per-request cache larger than the model weights for batch sizes above 4.
DeepSeek V4 uses MLA so the standard six factor formula understates the savings; per-token cache is closer to a single latent vector.
Qwen 3 70B-class deployments ship GQA-8; vLLM and SGLang both quote concurrent capacity off this exact arithmetic.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf your GPU has 80 GB and the model occupies 70 GB after sharding, how many simultaneous requests can you serve at 4k context with Llama-2 70B GQA?

QDoes the same formula apply to MLA (DeepSeek-V2)?

QHow much does int8 KV cache quantization save in practice, and what's the quality cost?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Forgetting the factor of 2 for K + V, using num_query_heads instead of num_kv_heads (matters under GQA/MQA), or using fp32 bytes (4) instead of fp16/bf16 (2).

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.