GQA = Grouped-Query Attention: Q heads are split into G groups that share one K and one V head per group, shrinking the KV cache by a factor of n_heads / G.
Imagine an office where every analyst (a query) has a personal filing cabinet (keys and values). Reading from thirty-two cabinets every time you decode a token is slow and expensive. Picture grouping the analysts into teams of eight, where each team shares a single cabinet. The analysts still bring their own questions to the table, but the lookup happens against one shared filing cabinet per team. You read far less from the cabinet room each step, and the answers stay almost as good as before. GQA is that shared-cabinet trick for attention, and it is why many open-weights chatbots run fast on the same hardware.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GQA is one of the most important quiet design choices in open-weights LLMs. The model cards for Llama 3.1, Qwen 3.5 and Gemma 4 list a query head count and a smaller KV head count side by side. That asymmetry is GQA at work, and it is the reason these models can run with long contexts on a single H100.
The story begins with multi-head attention as the original 2017 transformer (Vaswani et al.) defined it: every query head pairs with its own key head and value head, all the same count. That worked well for training but became a problem at inference time. Decoding a token from a long prompt needs a read of the entire KV cache from HBM on every step, and that cache grows linearly with the number of KV heads. By the time models reached 32 or 64 heads with multi-thousand-token contexts, the KV cache used most of the available memory bandwidth.
GQA is the structural fix. It keeps query heads at full count so the model's representational power is mostly intact, but it collapses K and V into a smaller set of shared heads.
The acronym and the basic shape
GQA expands to Grouped-Query Attention. The name describes exactly what changes versus multi-head attention. Query heads are partitioned into G equal groups. Inside a group, every query head shares a single key head and a single value head. Across groups, the K and V heads are independent.
Concretely, take Llama 3.1 8B. It has 32 query heads and 8 KV heads, which means G = 8 groups of 4 query heads each. Group 0's four queries all attend against KV pair 0. Group 1's four queries attend against KV pair 1. And so on. The query projections still run at full count, so each token still produces 32 distinct query vectors per layer.
The result is an attention layer that looks like MHA on the Q side and has a much smaller set of K and V heads on the other side. Implementation-wise, GQA is a one-line change in the attention kernel: the index used to gather K and V is query_head // (n_heads / G) rather than the query head index itself.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Variant | KV head count | KV cache size | Quality vs MHA |
|---|---|---|---|
| MHA | n_heads | Full | Baseline |
| GQA (G = 8) | G = 8 | Roughly 1/4 of MHA at 32 heads | Within a fraction of a percent |
| MQA | 1 | 1 / n_heads | Measurable regression on harder tasks |
Real products, models, and research that use this idea.
- Llama 3.1 8B ships GQA with 32 query heads and 8 KV heads, so each KV head serves 4 query heads.
- Llama 3.1 70B uses GQA with 64 query heads and 8 KV heads, so each KV head serves 8 query heads.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did so many models settle on G = 8 KV groups instead of 4 or 16?
Sweep the empirical curve: quality versus KV-cache reduction at G in {1, 2, 4, 8, 16, 32}. G = 8 captures most of the bandwidth win while quality stays within noise of MHA. More groups give too little savings; fewer groups move toward MQA and start to lower quality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying GQA reduces FLOPs. The savings are bandwidth and KV-cache memory, not compute. Q projections still run per head.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.