Drag each answer to line up with its matching prompt
MLA (DeepSeek)
Unique to MLA: an absorbed up projection matmul fuses into the attention path
GQA (Llama-3 style)
Single K and V head shared across all Q heads; max compression but quality regression on hard tasks
MQA (PaLM style)
MLA in DeepSeek-V3; GQA still dominant in most other open frontier models
Reconstruction cost
Structural sharing: multiple Q heads attend to one K/V head per group; G× cache reduction (G=8 typical)
Empirical 2026 winner
Low rank LATENT projection shared across heads; learned end to end; ~10× cache reduction vs MHA
MQA shares one KV head, GQA shares KV heads per group, and MLA stores a learned low-rank latent that reconstructs K and V on demand: three ways to shrink the KV cache.
Picture a library where every reader carries a full set of reference notes. That is multi-head attention: lots of duplicated notes, huge shelves. MQA says everyone share one single set of notes, cheapest but coarse. GQA says split readers into a few teams and give each team one shared set, a sensible middle ground. MLA says do not store the bulky notes at all, store a tiny compressed summary, then unpack the details only when a reader actually needs them. The summary fits in a drawer instead of a shelf. Unpacking costs a little extra work each time, but the storage savings are enormous, which is exactly the trade modern serving systems want.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-head Latent Attention, Grouped-Query Attention, and Multi-Query Attention are three answers to one question: how do you shrink the KV cache so that decode stops being memory-bandwidth bound? In full multi-head attention every query head carries its own key and value head, and the cache scales with the head count times the layer count times the sequence length. At long context that cache can exceed the model weights in HBM, and because decode reads the entire cache for every new token, the GPU spends its time streaming memory rather than computing.
The three methods sit on a spectrum but split into two families. MQA and GQA are structural: they keep real key and value heads, just fewer of them, shared across query heads. MLA is a compression scheme: it stores a learned low-rank latent per token and reconstructs the full keys and values on demand. Understanding that the families split, rather than treating the three as a single dial turned to different settings, is the entire point of this question.
This deep dive walks through each mechanism, the cache math, the quality tradeoffs, MLA's unique reconstruction cost and its RoPE complication, and the 2026 landscape of who uses what. By the end you should be able to match each method to its mechanism, its cache reduction factor, and its quality posture without hesitation.
The shared bottleneck: KV cache memory
During autoregressive decode the attention layer at every level reads the keys and values of all prior tokens. Those are cached so they are not recomputed, and the cache size follows a simple formula.
Here L is layers, H_kv is the number of KV heads, d_h is head dimension, T is sequence length, and b is bytes per element. The factor of 2 counts keys and values. The single lever all three methods pull is H_kv, the KV head count. Full multi-head attention sets H_kv equal to the query head count. Every method below drives H_kv down, or in MLA's case replaces the per-head storage entirely with a much smaller latent dimension.
The reason this matters is bandwidth, not compute. Decode reads the whole cache per token, so halving the cache nearly halves decode latency. That is why frontier models stopped shipping plain multi-head attention years ago.
It also reframes what these three methods are doing. None of them changes the attention math the model learns; they change how the key and value information is stored and retrieved. MQA and GQA cut H_kv by hard-wiring fewer physical heads. MLA leaves H_kv conceptually intact but stores a compressed surrogate, then expands it. Keeping that distinction in mind is what lets you reason about quality and runtime cost rather than just memorizing reduction factors.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Method | What is stored | Cache reduction | Quality posture | Runtime catch |
|---|---|---|---|---|
| MHA | Full K and V per head | 1x baseline | Reference quality | Largest cache, decode is bandwidth-bound |
| MQA | One shared K and V head | n_heads x | Regresses on hard tasks | Coarsest single history view |
| GQA | One K and V head per group | G x (G=8 typical) | Near full quality | None beyond standard attention |
| MLA | Low-rank latent per token | ~10x | Matches GQA | Up-projection matmul, decoupled RoPE |
Real products, models, and research that use this idea.
- DeepSeek-V3 ships Multi-head Latent Attention to cut per-request cache about tenfold versus full multi-head attention while matching GQA-level quality.
- Llama-3 and Llama 4 use Grouped-Query Attention with 8 KV groups, the de facto default across most open frontier models in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does MQA regress on hard reasoning tasks while GQA largely does not?
Think about representational capacity. One shared KV head forces all query heads through a single view of history, collapsing head diversity. GQA keeps several KV heads, preserving enough distinct views that quality holds while still cutting the cache by the group count.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating MLA as just aggressive GQA. MLA stores a low-rank latent and reconstructs K and V at runtime; GQA stores real shared heads with no reconstruction step.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.