Match each attention variant to its KV-cache property and a production user
MHA keeps one KV head per query head (biggest cache, top quality), MQA shares a single KV head (smallest cache, some quality loss), and GQA groups query heads to share KV heads: the practical middle.
Imagine a meeting room where many note-takers each summarize the discussion. In MHA, every note-taker also keeps their own private filing cabinet of source documents. That is great quality, but the cabinets fill the whole building. In MQA, everyone shares one filing cabinet. Tiny footprint, but with only one cabinet some nuance gets lost and answers get a little worse. GQA is the compromise: split the note-takers into a few teams, and each team shares one cabinet. You shrink the storage a lot while keeping almost all the quality, because each team still has its own well-organized records. Modern systems pick the team setup because it saves space without making the work noticeably worse.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: name the shared head in each variant + map KV-head count onto the cache formula + the quality versus memory trade + why G=8 won + which production models ship which variant.
| Property | MHA | GQA (G=8) | MQA |
|---|---|---|---|
| KV heads | equal to query heads | query heads divided by 8 | exactly 1 |
| Cache size | 1x (baseline) | about 1/8 | 1 over head count |
| Quality | reference, best | near MHA after uptraining | noticeable regression |
| Decode compute | baseline | essentially same | essentially same |
| Production use in 2026 | small or legacy models | Llama, Mistral, Qwen default | PaLM, Falcon legacy |
Real products, models, and research that use this idea.
- Llama 4, Mistral Large 3, and Qwen 3 all ship Grouped-Query Attention at group size eight as their production default in 2026.
- Multi-Query Attention, the most aggressive single-KV-head design, survives in 2026 mainly inside latency-critical edge and high-QPS chat stacks where KV memory is the hard limit.
- Cross-layer KV sharing in recent efficiency-focused open models pushes reduction past GQA by reusing the cache across layers, not just within one.
- Llama-2-70B popularized GQA at scale; its 7B and 13B siblings stayed on MHA, making it a clean side-by-side comparison.
- DeepSeek V4 moved past GQA to Multi-head Latent Attention, compressing KV to a low-rank latent for even smaller per-request cache.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does reducing KV heads barely hurt quality while reducing query heads would?
QHow do you convert an existing MHA checkpoint into a GQA model?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying GQA and MQA shrink the cache by reducing query heads. They reduce KV heads only; the query heads stay at the full count, so model capacity is largely preserved.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.