Compare MHA, MQA, GQA, MLA, what production tradeoff are they all addressing, and which models use each?
Compare the four attention variants: MHA, MQA, GQA, MLA. What's the production bottleneck they all target, and which real world models use each?
All four shrink KV cache. MHA: per-head K/V. MQA: one shared. GQA: G groups (Llama-3, Mistral). MLA: low rank latent (DeepSeek V3/V4).
Picture an LLM as a librarian helping you with a long conversation. The model itself is the librarian's training, what they know. The KV cache is the stack of sticky notes the librarian keeps next to your conversation, one note per word so they can look back. As the chat gets longer, that stack of notes can grow bigger than the librarian's own knowledge, and at some point the desk runs out of room. MHA, MQA, GQA, and MLA are four progressively cleverer ways to shrink the sticky note stack without making the librarian forget too much of what you said. Each is a different trade between desk space and how well the librarian still answers.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MHA, MQA, GQA, and MLA are not four parallel attention mechanisms competing on different dimensions. They are four points on a single curve, each addressing the same production constraint at a different trade between compression and quality. That constraint is KV cache memory.
Why the same constraint keeps producing new variants is itself the lesson worth teaching. Context lengths grow. Batch sizes grow. Agentic and multi-tenant workloads multiply both. Every doubling of context or batch doubles the cache. Model weights have a fixed memory footprint per parameter; the cache scales linearly with how much you've talked to the model. At long context with moderate batch, cache memory exceeds weight memory, and the cache becomes the binding constraint on what hardware can serve at what cost.
We'll walk the cache formula, then take each variant in turn, then close with the practical 2026 picture.
The common target: KV cache memory
Cache memory per request follows:
The factor of 2 covers K and V. L is layer count, H_kv is the number of KV heads, d_h is head dimension, T is sequence length, b is bytes per element.
All four variants attack H_kv, either by reducing it directly (MQA, GQA) or by changing what's stored in the cache slot at all (MLA). None of them touch L, d_h, T, or b, because those are architectural commitments or runtime properties that the attention design can't move.
That focus on one variable is what makes the comparison clean. Pick a variant, plug in the new H_kv, and you have the new cache footprint.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Variant | What's shared | Cache savings vs MHA | Quality | Production users |
|---|---|---|---|---|
| MHA | Nothing, per-head K, V | 1x baseline | Best | GPT-2/3, Llama-1, BERT |
| MQA | All heads share ONE K, V | Nx (often 32-64x) | Drops on some tasks | PaLM, Falcon-1, StarCoder |
| GQA | G groups share K, V | N/G x (typically 8x) | Near-MHA | Llama-2/3, Mistral, Mixtral, Qwen, Gemma |
| MLA | Low-rank latent per token | ≈ 10-20x or more | Matches MHA | DeepSeek-V2/V3 |
Real products, models, and research that use this idea.
- Llama 4 Maverick, Mistral Large 3, Qwen 3, Gemma 4: GQA in 2026 production.
- Llama-2 70B and Llama-3 70B: GQA with 8 KV heads on 64 query heads.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through MLA's matmul absorption trick, how does the decompression avoid runtime cost?
K = c_t × W_K_d, then attention dot Q × K = Q × (c_t × W_K_d) = (Q × W_K_d^T) × c_t. So if Q is right-multiplied by W_K_d^T once at projection time, the attention kernel works directly on c_t. Same for V → O via O projection. The latent never has to be expanded back to per-head K, V at attention time.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the four as just 'different attention mechanisms' rather than understanding they're all responses to the same KV-cache-memory bottleneck, each is a different trade between cache size and expressiveness.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
- Vaswani et al. 2017 — Attention Is All You Need (MHA)
- Shazeer 2019 — Fast Transformer Decoding: One Write-Head is All You Need (MQA)
- Ainslie et al. 2023 — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-AI 2024 — DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (MLA)
Same topic, related formats. Practice these next.