GQA gives a tunable knob: share K and V across groups of query heads to shrink the KV cache like MQA, while keeping enough KV heads to hold MHA-level quality.
Picture a classroom where every student (a query head) needs to consult reference notes before answering. Multi-head attention gives each student a personal, hand-written copy of the notes, which is accurate but uses a huge amount of paper. Multi-query attention hands the whole class a single shared copy, saving paper but causing crowding and mistakes when the material gets hard. Grouped-query attention splits the class into small groups, say eight students per shared copy. You still save most of the paper, but each group's notes stay specialized enough that answers barely suffer. That middle setting is why many open language models pick it: nearly the savings of one shared copy, nearly the quality of personal copies.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The KV cache is the main cost of language-model decoding. Multi-head attention, multi-query attention and grouped-query attention each set how much of that cache a model keeps, and how much quality it gives up for a smaller one.
All three compute attention the same way. They differ only in how many distinct key and value heads they keep. That single design choice changes the size of the KV cache. The cache size in turn sets how many bytes the GPU must stream from high-bandwidth memory on every decode step. Because decode is bandwidth-bound rather than compute-bound, shrinking the cache raises throughput. A model that halves its cache can roughly double the number of concurrent requests it serves on the same hardware, which lowers cost per token.
The wrong options name compute cost, parameter count and the direction of the cache change. None of these is the axis that matters. The correct option names the memory axis and the mechanism: a tunable group size that recovers MHA quality while giving the cache reduction MQA offered. A group size near eight became common, an existing model can be converted cheaply, and newer compression schemes extend the same idea.
The three variants, defined by KV head count
Start from multi-head attention. With n_heads query heads, MHA also maintains n_heads key heads and n_heads value heads. Each query head attends through its own private key and value, so every head can learn to look for a different kind of relationship. This gives maximum expressiveness and the largest cache.
Multi-query attention keeps all the query heads but collapses to a single key head and a single value head shared across every query. The query side is unchanged, so the model still has the same number of attention computations. But the cache now stores keys and values for one head instead of many. All sixty-four query heads in a typical model now read from the same key, which is where the capacity pressure comes from.
Grouped-query attention generalizes both. You pick an intermediate KV head count. Each KV head is shared by a contiguous group of query heads, so a group of eight query heads might share one key and one value head. Set the KV head count to one and you have MQA. Set it equal to n_heads and you have MHA. Everything in between is a valid GQA configuration, which is the tunability the correct option names. The single integer that picks the operating point is what makes the family a continuum rather than three separate designs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.1 70B ships GQA with eight KV heads, a common production setting that balances cache size against quality.
- Llama 3 8B uses grouped-query attention, with 32 query heads sharing 8 KV heads, so long-context decode stays within HBM budgets on a single node.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does sharing KV heads barely change compute even though it cuts memory?
Trace the matmuls. Every query head still computes its own scores against the shared key, so the number of query times key dot products is unchanged. What shrinks is the size of the K and V tensors held in HBM and streamed each decode step.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming GQA wins on compute or parameter count. The win is KV-cache bytes and bandwidth. The query-side matmul is unchanged, and only the K and V projections shrink.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.