GQA = Grouped-Query Attention: Q heads are split into G groups that share one K and one V head per group, shrinking the KV cache by the group factor.
Imagine an office where every analyst (a query) has a personal filing cabinet (keys and values). Reading from twenty cabinets every time you decode a token is slow and expensive. Picture grouping the analysts into teams of eight, where each team shares a single cabinet. The analysts still bring their own questions to the table, but the lookup happens against one shared filing cabinet per team. You read far less from the cabinet room each step, and the answers stay almost as good as before. GQA is that shared-cabinet trick for attention, and it is why modern open-weights chatbots feel fast on the same hardware.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
GQA is one of the most quietly important architectural choices in modern open-weights LLMs. If you open any 2026 model card for Llama 3.1, Mistral Large 3, Qwen 3.5, or Gemma 4, you will see a query head count and a smaller KV head count printed side by side. That asymmetry is GQA at work, and it is the reason these models can run with long contexts on a single H100 instead of needing exotic memory tricks.
The story begins with multi-head attention as Vaswani's original 2017 transformer defined it: every query head pairs with its own key head and value head, all the same count. That worked beautifully for training but became a problem at inference time. Decoding a token from a long prompt requires reading the entire KV cache from HBM on every step, and that cache grows linearly with the number of KV heads. By the time models reached 32 or 64 heads with multi-thousand-token contexts, the KV cache was eating most of the available memory bandwidth.
GQA is the structural fix. It keeps query heads at full count so the model's representational power is mostly intact, but it collapses K and V into a smaller set of shared heads. This deep dive walks through the mechanics, the bandwidth argument, the group-size knob, the comparison with MHA and MQA, and the 2026 deployment patterns where you will encounter it.
The acronym and the basic shape
GQA expands to Grouped-Query Attention. The name describes exactly what changes versus multi-head attention. Query heads are partitioned into G equal groups. Inside a group, every query head shares a single key head and a single value head. Across groups, the K and V heads are independent.
Concretely, take Llama 3.1 8B. It has 32 query heads and 8 KV heads, which means G = 8 groups of 4 query heads each. Group 0's four queries all attend against KV pair 0. Group 1's four queries attend against KV pair 1. And so on. The query projections still run at full count, so each token still produces 32 distinct query vectors per layer.
The result is an attention layer that looks like MHA on the Q side and a much skinnier K and V tier on the other side. Implementation-wise, GQA is a one-line change in the attention kernel: the index used to gather K and V is query_head // (n_heads / G) rather than the query head index itself.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Variant | KV head count | KV cache size | Quality vs MHA |
|---|---|---|---|
| MHA | n_heads | Full | Baseline |
| GQA (G = 8) | G = 8 | Roughly 1/4 of MHA at 32 heads | Within a fraction of a percent |
| MQA | 1 | 1 / n_heads | Measurable regression on harder tasks |
Real products, models, and research that use this idea.
- Llama 3.1 8B ships GQA with 32 query heads and 8 KV heads, the canonical 4:1 ratio used across the Llama 3 family.
- Mistral Large 3 uses GQA at group size 8 to keep the KV cache small enough for long-context serving on H100 fleets.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did the field converge on group size 8 instead of 4 or 16?
Sweep the empirical curve: quality versus KV-cache reduction at G in {1, 2, 4, 8, 16, 32}. Group 8 captures most of the bandwidth win while quality stays within noise of MHA. Smaller groups give too little savings; larger groups push toward MQA territory and start to bite into quality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying GQA reduces FLOPs. The savings are bandwidth and KV-cache memory, not compute. Q projections still run per head.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.