Drag each answer to line up with its matching prompt
MHA (Multi-Head Attention)
K_heads = Q_heads / G: tunable, near MHA quality at G=8
MQA (Multi-Query Attention)
K_heads = 1: maximum cache compression, noticeable quality dip
GQA (Grouped Query Attention)
K_heads = Q_heads: full KV cache, highest quality, highest memory
Llama-3 / Mistral / Llama-2-70B
Production default GQA with 8 KV heads
Original PaLM / Falcon-7B
Early production deployments of MQA
MHA keeps one KV head per query head (biggest cache, top quality), MQA shares a single KV head (smallest cache, some quality loss), and GQA groups query heads to share KV heads: the practical middle.
Imagine a meeting room where many note-takers each summarize the discussion. In MHA, every note-taker also keeps their own private filing cabinet of source documents. That is great quality, but the cabinets fill the whole building. In MQA, everyone shares one filing cabinet. Tiny footprint, but with only one cabinet some nuance gets lost and answers get a little worse. GQA is the compromise: split the note-takers into a few teams, and each team shares one cabinet. You shrink the storage a lot while keeping almost all the quality, because each team still has its own well-organized records. Most open models pick the team setup because it saves space without making the work noticeably worse.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-Head, Multi-Query, and Grouped-Query Attention are three points on a single design axis: how many distinct key and value heads a layer keeps. The KV cache is the dominant memory cost during decode, and the KV-head count is the one setting that directly sets that cost.
All three variants keep the full set of query heads. They never change the query side. What they vary is how many key heads and value heads exist for those queries to attend over. MHA gives each query head its own KV head. MQA gives every query head a single shared KV head. GQA groups query heads so each group shares one KV head, landing in between.
The cache formula ties KV-head count to memory, and sharing has a quality cost. Eight KV heads became a common default, and production models moved from MHA to MQA to GQA. With the formula, a model's config is enough to place it on the spectrum and predict its cache footprint.
The shared axis: how many KV heads
Self-attention projects each token into queries, keys, and values, splitting them across heads. In the original design each head is fully independent: query head i attends using key head i and value head i. With sixty-four heads you store sixty-four sets of keys and values per token, one per head, at every layer.
The three variants differ only in the count of distinct KV heads. Multi-Head Attention keeps one KV head per query head, so the KV-head count equals the query-head count. Multi-Query Attention keeps exactly one KV head shared by all query heads. Grouped-Query Attention partitions the query heads into groups and gives each group its own KV head, so the KV-head count is the query-head count divided by the group size.
The query heads are identical across all three. You are never reducing the number of attention patterns the layer can express. You are reducing only the number of distinct key and value projections those patterns read from, which is precisely the data that must persist in the cache.
Mechanically, the shared KV head is broadcast across the query heads in its group during the attention matmul. Each query head still computes its own scores against the shared keys and its own weighted sum over the shared values. So the number of dot products is unchanged; only the number of distinct key and value tensors that must be materialized and stored goes down. That distinction is why the saving is purely a memory and bandwidth win, not a compute win.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | MHA | GQA (G=8) | MQA |
|---|---|---|---|
| KV heads | equal to query heads | query heads divided by 8 | exactly 1 |
| Cache size | 1x (baseline) | about 1/8 | 1 over head count |
| Quality | reference, best | near MHA after uptraining | noticeable regression |
| Decode compute | baseline | about the same | about the same |
| Production use | small or legacy models | Llama 3, Mistral 7B, Qwen3 | PaLM, Falcon-7B (legacy) |
Real products, models, and research that use this idea.
- Llama 4 Scout ships Grouped-Query Attention with 40 query heads and 8 KV heads, and Qwen3-235B-A22B pairs 64 query heads with 4 KV heads.
- Multi-Query Attention, the single-KV-head design, ships in Falcon-7B, StarCoder and the Gemma 1 2B checkpoints.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does reducing KV heads barely hurt quality while reducing query heads would?
Query heads define how many distinct attention patterns the layer can express, so cutting them removes capacity. KV heads only supply the shared keys and values those patterns read; several query heads can attend over one shared KV head with little loss, especially after a short uptraining phase.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying GQA and MQA shrink the cache by reducing query heads. They reduce KV heads only; the query heads stay at the full count, so model capacity is largely preserved.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.