Compare the four attention variants: MHA, MQA, GQA, MLA. What's the production bottleneck they all target, and which real-world models use each?
Free · 2 AI evals / day
All four are attention mechanisms that differ in how K and V are shared across query heads. They all exist for one production reason: shrinking the KV cache.
**MHA** is the original. N query heads, N key heads, N value heads. Llama-2 70B at 4k context with naive MHA produces ~10 GB of cache per request. Best quality, biggest cache.
**MQA** collapses K and V to one shared head. N queries dot against the same K and weight the same V. Cache shrinks N-fold. Quality drops measurably on harder tasks. PaLM and Falcon used it.
**GQA** is the compromise the field settled on. Split N query heads into G groups; each group shares one K, V. Llama 4 Maverick uses G=8 on 64 query heads, 8x cache cut at near-MHA quality. Mistral Large 3, Qwen 3, Gemma 4 all use GQA.
**MLA** (DeepSeek V2, 2024) compresses K, V into a per-token latent (~512 dim). Cache stores only the latent; decompression absorbs into Q and O projections. Smaller cache than MQA at MHA matching quality. DeepSeek V2/V3/V4.
All four target **KV cache memory**, which dominates LLM serving cost as context length and batch size grow. KV cache memory scales as:
$$\text{bytes} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot b$$
Each variant attacks the `H_kv` factor differently.
- MHA: per-head K and V. Baseline. Best quality, biggest cache. Used in GPT-2/3, Llama-1, BERT.
- MQA: one shared K and V across all query heads. Max compression but measurable quality drop. PaLM and Falcon used it; mostly historical now.
- GQA: K and V shared across `G` groups. The production sweet spot. Near-MHA quality at roughly 8x cache reduction. Llama-3, Llama 4 Maverick, Mistral Large 3, Qwen 3, Gemma 4.
- MLA (DeepSeek V2/V3/V4): K and V compressed into a low rank latent around 512 dim. Cache smaller than MQA's, quality matches MHA.
The progression is the iterative response to context length serving pressure. GQA is today's default. MLA is the live frontier.
Quick reference
TL;DR
All four shrink KV cache. MHA: per-head K/V. MQA: one shared. GQA: G groups (Llama-3, Mistral). MLA: low rank latent (DeepSeek V3/V4).
Picture an LLM as a librarian helping you with a long conversation. The model itself is the librarian's training, what they know. The KV cache is the stack of sticky notes the librarian keeps next to your conversation, one note per word so they can look back. As the chat gets longer, that stack of notes can grow bigger than the librarian's own knowledge, and at some point the desk runs out of room. MHA, MQA, GQA, and MLA are four progressively cleverer ways to shrink the sticky note stack without making the librarian forget too much of what you said. Each is a different trade between desk space and how well the librarian still answers.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Encoder-only or short context classification models Without an autoregressive KV cache, the cache-compression motivation disappears and MHA quality is hard to beat.
Tiny models (<1B params) where cache is already small Cache memory isn't the binding constraint at this scale; the quality-compression trade isn't worth the complexity.
Frameworks without GQA / MLA kernel support Hand-rolling these inside a stock attention kernel destroys the throughput advantage; only adopt when fused kernels exist.
Architectures where positional encoding is non-trivially coupled to K MLA needs decoupled RoPE; bolting MLA on top of a position scheme that doesn't separate cleanly degrades quality.
Quick scan · 2–4 min
Everything important, quickly.
Walk the cache formula's dependence on num_kv_heads, position each variant on the sharing spectrum, quote savings ratios with concrete model examples, explain GQA uptraining from MHA, and detail MLA's latent + decoupled RoPE. Connect to the industry arc.
Side-by-side
Variant
What's shared
Cache savings vs MHA
Quality
Production users
MHA
Nothing, per-head K, V
1x baseline
Best
GPT-2/3, Llama-1, BERT
MQA
All heads share ONE K, V
Nx (often 32-64x)
Drops on some tasks
PaLM, Falcon-1, StarCoder
GQA
G groups share K, V
N/G x (typically 8x)
Near-MHA
Llama-2/3, Mistral, Mixtral, Qwen, Gemma
MLA
Low-rank latent per token
≈ 10-20x or more
Matches MHA
DeepSeek-V2/V3
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick, Mistral Large 3, Qwen 3, Gemma 4: GQA in 2026 production.
2Llama-2 70B and Llama-3 70B: GQA with 8 KV heads on 64 query heads.
5DeepSeek V2, V3, and V4: MLA with latent dim around 512.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through MLA's matmul absorption trick, how does the decompression avoid runtime cost?
Sign in or go Pro to see the approach for this follow-up.
QWhy is GQA preferred over MQA in modern open models even though MQA has bigger cache savings?
Sign in or go Pro to see the approach for this follow-up.
QHow does MLA interact with positional encodings (RoPE)?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the four as just 'different attention mechanisms' rather than understanding they're all responses to the same KV-cache-memory bottleneck, each is a different trade between cache size and expressiveness.