GQA = Grouped-Query Attention. Query heads partition into G groups; each group shares one K and one V head. Llama-2 70B uses 64 query heads on 8 KV heads.
Imagine 64 people on a quiz team, each holding a different question. In regular multi-head attention, every person gets a personal librarian and a personal bookshelf. That is expensive: 64 librarians, 64 shelves. Grouped-query attention says: split the 64 people into 8 groups of 8, and each group shares one librarian and one shelf. The 8 people in a group ask different questions, but they all consult the same librarian for answers. You go from 64 shelves down to 8, an 8x storage saving, and the team's quiz scores barely drop because the librarians were duplicating each other's work anyway.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GQA, Grouped-Query Attention, is the production answer to the KV cache crisis at LLM scale. The KV cache stores K and V for every past token per head per layer, and at 70B+ scale with long context, that cache dominates inference memory and bandwidth. GQA shrinks it by sharing K and V across groups of query heads.
The design is one of those rare proposals that combines large memory savings with negligible quality cost, which is why it became the default for production LLMs almost immediately after the 2023 GQA paper. This deep dive walks the mechanism, why MHA was wasteful in the first place, the KV cache math that makes GQA important, the empirical ablations behind Llama-2 70B's specific choice, and the modern landscape where GQA is one option on a broader KV-compression axis.
The mechanism, head by head
GQA partitions the num_heads query heads into G groups, each sharing one K head and one V head. The group size is G = num_heads / num_kv_heads (the two must divide evenly).
What changes versus MHA
- The Q projection produces
num_heads * d_headoutputs, same as MHA. - The K and V projections produce
num_kv_heads * d_headoutputs, smaller than MHA. - Inside a group, the query heads compute different attention scores against the same K, then take a weighted sum over the same V.
- The output projection mixes across all query head outputs as usual.
What stays the same
- The attention math itself:
softmax(QK^T / sqrt(d_k)) Vper query head against its assigned K, V. - The per-token output shape.
- The interaction with positional encoding, FlashAttention, masks, dropout, everything downstream.
The slider
GQA is the general case; MHA and MQA are the endpoints:
- G = 1: every query head has its own KV. This is plain MHA.
- G = num_heads: all query heads share one KV head. This is MQA.
- 1 < G < num_heads: GQA proper.
Llama-2 70B picks G = 8 with num_heads = 64, num_kv_heads = 8. Mistral Large 3 picks similar ratios. The specific choice is per-model and empirically tuned.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama-2 70B: 64 query heads, 8 KV heads, group size 8. The original GQA production deployment.
- Llama 4 Maverick and Mistral Large 3 inherit Llama-2's GQA pattern for KV cache reduction at scale.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does GQA cost so little quality compared to plain MHA?
Empirically, K and V representations in trained MHA models exhibit substantial redundancy across heads. Many heads attend to similar patterns, with per-head specialization concentrated more in the Q projection and the output mixing. Sharing K and V across a small group exploits this redundancy without sacrificing what makes attention work. Meta's ablations confirm the perplexity hit at G = 8 is in the noise on standard benchmarks.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking GQA is just MQA with extra heads. Both share KV across queries, but GQA picks a group size between 1 (MHA) and num_heads (MQA) to balance memory savings against quality.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.