Drag each answer to line up with its matching prompt
MHA (Multi-Head Attention)
Production default GQA with group size 8
MQA (Multi-Query Attention)
K_heads = Q_heads: full KV cache, highest quality, highest memory
GQA (Grouped Query Attention)
Early production deployments of MQA
Llama-3 / Mistral / Llama-2-70B
K_heads = 1: maximum cache compression, noticeable quality dip
Original PaLM / Falcon-40B
K_heads = Q_heads / G: tunable, near MHA quality at G=8
MHA keeps one KV head per query head (biggest cache, top quality), MQA shares a single KV head (smallest cache, some quality loss), and GQA groups query heads to share KV heads: the practical middle.
Imagine a meeting room where many note-takers each summarize the discussion. In MHA, every note-taker also keeps their own private filing cabinet of source documents. That is great quality, but the cabinets fill the whole building. In MQA, everyone shares one filing cabinet. Tiny footprint, but with only one cabinet some nuance gets lost and answers get a little worse. GQA is the compromise: split the note-takers into a few teams, and each team shares one cabinet. You shrink the storage a lot while keeping almost all the quality, because each team still has its own well-organized records. Modern systems pick the team setup because it saves space without making the work noticeably worse.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-Head, Multi-Query, and Grouped-Query Attention are three points on a single design axis: how many distinct key and value heads a layer keeps. Every modern serving interview circles back to this axis, because the KV cache is the dominant memory cost during decode, and the KV-head count is the one knob that directly sets that cost.
The critical thing to internalize is that all three variants keep the full set of query heads. They never touch the query side. What they vary is how many key heads and value heads exist for those queries to attend over. MHA gives each query head its own KV head. MQA gives every query head a single shared KV head. GQA groups query heads so each group shares one KV head, landing in between.
This deep dive walks through the mechanism behind each variant, the exact cache math that ties KV-head count to memory, the quality cost of sharing, why group size eight emerged as the practical default, and the production timeline that moved the field from MHA to MQA to GQA. By the end you should be able to place any model on the spectrum and predict its cache footprint from its config.
The shared axis: how many KV heads
Self-attention projects each token into queries, keys, and values, splitting them across heads. In the original design each head is fully independent: query head i attends using key head i and value head i. With sixty-four heads you store sixty-four sets of keys and values per token, one per head, at every layer.
The three variants differ only in the count of distinct KV heads. Multi-Head Attention keeps one KV head per query head, so the KV-head count equals the query-head count. Multi-Query Attention keeps exactly one KV head shared by all query heads. Grouped-Query Attention partitions the query heads into groups and gives each group its own KV head, so the KV-head count is the query-head count divided by the group size.
The query heads are identical across all three. That is the point people miss. You are never reducing the number of attention patterns the layer can express. You are reducing only the number of distinct key and value projections those patterns read from, which is precisely the data that must persist in the cache.
Mechanically, the shared KV head is broadcast across the query heads in its group during the attention matmul. Each query head still computes its own scores against the shared keys and its own weighted sum over the shared values. So the number of dot products is unchanged; only the number of distinct key and value tensors that must be materialized and stored goes down. That distinction is why the saving is purely a memory and bandwidth win, not a compute win.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | MHA | GQA (G=8) | MQA |
|---|---|---|---|
| KV heads | equal to query heads | query heads divided by 8 | exactly 1 |
| Cache size | 1x (baseline) | about 1/8 | 1 over head count |
| Quality | reference, best | near MHA after uptraining | noticeable regression |
| Decode compute | baseline | essentially same | essentially same |
| Production use in 2026 | small or legacy models | Llama, Mistral, Qwen default | PaLM, Falcon legacy |
Real products, models, and research that use this idea.
- Llama 4, Mistral Large 3, and Qwen 3 all ship Grouped-Query Attention at group size eight as their production default in 2026.
- Multi-Query Attention, the most aggressive single-KV-head design, survives in 2026 mainly inside latency-critical edge and high-QPS chat stacks where KV memory is the hard limit.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does reducing KV heads barely hurt quality while reducing query heads would?
Query heads define how many distinct attention patterns the layer can express, so cutting them removes capacity. KV heads only supply the shared keys and values those patterns read; several query heads can attend over one shared KV head with little loss, especially after a short uptraining phase.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying GQA and MQA shrink the cache by reducing query heads. They reduce KV heads only; the query heads stay at the full count, so model capacity is largely preserved.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.