Compare MHA, MQA, GQA, MLA, what production tradeoff are they all addressing, and which models use each?
Compare the four attention variants: MHA, MQA, GQA, MLA. What's the production bottleneck they all target, and which real-world models use each?
All four shrink KV cache. MHA: per-head K/V. MQA: one shared. GQA: G groups (Llama-3, Mistral). MLA: low rank latent (DeepSeek V3/V4).
Picture an LLM as a librarian helping you with a long conversation. The model itself is the librarian's training, what they know. The KV cache is the stack of sticky notes the librarian keeps next to your conversation, one note per word so they can look back. As the chat gets longer, that stack of notes can grow bigger than the librarian's own knowledge, and at some point the desk runs out of room. MHA, MQA, GQA, and MLA are four progressively cleverer ways to shrink the sticky note stack without making the librarian forget too much of what you said. Each is a different trade between desk space and how well the librarian still answers.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Walk the cache formula's dependence on num_kv_heads, position each variant on the sharing spectrum, quote savings ratios with concrete model examples, explain GQA uptraining from MHA, and detail MLA's latent + decoupled RoPE. Connect to the industry arc.
| Variant | What's shared | Cache savings vs MHA | Quality | Production users |
|---|---|---|---|---|
| MHA | Nothing, per-head K, V | 1x baseline | Best | GPT-2/3, Llama-1, BERT |
| MQA | All heads share ONE K, V | Nx (often 32-64x) | Drops on some tasks | PaLM, Falcon-1, StarCoder |
| GQA | G groups share K, V | N/G x (typically 8x) | Near-MHA | Llama-2/3, Mistral, Mixtral, Qwen, Gemma |
| MLA | Low-rank latent per token | ≈ 10-20x or more | Matches MHA | DeepSeek-V2/V3 |
Real products, models, and research that use this idea.
- Llama 4 Maverick, Mistral Large 3, Qwen 3, Gemma 4: GQA in 2026 production.
- Llama-2 70B and Llama-3 70B: GQA with 8 KV heads on 64 query heads.
- PaLM, Falcon-1, StarCoder: MQA (mostly historical now).
- Original GPT-2/3, Llama-1, BERT, T5: MHA baseline.
- DeepSeek V2, V3, and V4: MLA with latent dim around 512.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through MLA's matmul absorption trick, how does the decompression avoid runtime cost?
QWhy is GQA preferred over MQA in modern open models even though MQA has bigger cache savings?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating the four as just 'different attention mechanisms' rather than understanding they're all responses to the same KV-cache-memory bottleneck, each is a different trade between cache size and expressiveness.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
- Vaswani et al. 2017 — Attention Is All You Need (MHA)
- Shazeer 2019 — Fast Transformer Decoding: One Write-Head is All You Need (MQA)
- Ainslie et al. 2023 — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-AI 2024 — DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (MLA)
Same topic, related formats. Practice these next.