Same topic, related formats. Practice these next.
Derive the KV cache memory formula. Why does it (rather than model weights) become the dominant memory cost at long context, and how does GQA mitigate this?
Weights are flat; KV cache scales with context times batch and surpasses weights fast. GQA cuts kv_heads for integer savings, Llama-3 70B drops 64 heads to 8 KV heads for 8x.
Picture the model as a fixed encyclopedia, same shelf space whether you ask one question or a thousand. The model's working notes (the KV cache) are different: every word in every active conversation adds a new note, and the stack of notes grows with both how long each chat gets and how many chats are happening at once. Past a certain context length, the notes take more room than the encyclopedia. GQA shrinks each note by letting groups of readers share one set of K and V slots, Llama-3 70B uses 8 shared sets across 64 readers, so the note pile shrinks 8x.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Derive the formula, contrast with the fixed weight memory, walk a concrete crossover example, explain GQA's mechanism and savings ratio, place it in the MHA→MQA→GQA→MLA arc, and tie to production levers (paged allocation, KV quantization, prefix sharing).
def cache_vs_weights(num_params_billion, num_layers, num_kv_heads, d_head,
bytes_per_value, seq_len, batch_size):
weights_gb = num_params_billion * bytes_per_value
per_req_cache_gb = (2 * num_layers * num_kv_heads * d_head
* bytes_per_value * seq_len) / 1e9
total_cache_gb = per_req_cache_gb * batch_size
crossover = total_cache_gb / weights_gb
return dict(weights_gb=weights_gb,
per_req_cache_gb=round(per_req_cache_gb, 2),
total_cache_gb=round(total_cache_gb, 2),
cache_to_weight_ratio=round(crossover, 2))
# Llama-3 70B, fp16, 32k context, batch 16, GQA-8
print(cache_vs_weights(70, 80, 8, 128, 2, 32768, 16))
# weights ~140 GB, total cache ~172 GB — cache exceeds weights
# Same model, but textbook MHA (64 KV heads), batch 16, 32k
print(cache_vs_weights(70, 80, 64, 128, 2, 32768, 16))
# total cache ~1376 GB — completely infeasible| Variant | num_kv_heads (Llama-2 70B shape) | Cache savings vs MHA | Quality impact |
|---|---|---|---|
| MHA | 64 (= num_query_heads) | 1x baseline | baseline |
| MQA | 1 | 64x | Noticeable on some tasks |
| GQA (G=8) | 8 | 8x | Near-MHA, production standard |
| MLA | n/a (low rank latent ~ 512 dim) | ≈10-20x | Comparable to MHA per DeepSeek-V2 |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Believing model weights are the binding memory constraint at long context, they're fixed; KV cache is what scales, and it surpasses weights quickly at moderate context × batch.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.