For a 64-Q-head model with GQA group size G=8, how many K/V heads exist and what is the cache reduction vs MHA?
A transformer has 64 query heads. It uses GQA with group size G=8. How many K and V heads does it have? What is the KV-cache reduction factor relative to plain MHA at the same head_dim and sequence length? Does it cost extra FLOPs at inference time?
K and V heads equal 64/8 = 8, so the KV cache shrinks by exactly the group size G = 8x versus MHA, with zero extra inference FLOPs.
Picture 64 readers in a library, each needing a reference book. Under plain attention every reader gets a private copy, so the shelf holds 64 books. GQA arranges the readers into 8 teams of 8, and each team shares one book. Now the shelf holds only 8 books, eight times less storage. The readers still read independently and ask their own questions, so no reader does extra work. The only change is that teammates look at the same shared book instead of personal copies. You keep the same number of readers and the same reading effort, but you slash the shelf space. That shelf is the KV cache, and shrinking it is what lets you serve long contexts without running out of memory.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Grouped-query attention is one of the cleanest pieces of arithmetic in inference optimization, which is exactly why interviewers like it. The numbers are small enough to do in your head, but the reasoning exposes whether you understand where the KV cache cost actually lives. The question here is concrete: 64 query heads, group size 8, so how many KV heads, what cache reduction, and is there a FLOP penalty?
The short answers are 8 KV heads, an 8 times cache reduction, and no FLOP penalty. But the value is in justifying each one. The KV head count comes from a single division. The reduction factor follows from which term of the cache formula GQA touches. The lack of a FLOP penalty follows from counting dot products at attention time.
This deep dive derives all three carefully, then explains why the real prize is HBM bandwidth rather than memory capacity, and where GQA sits between full multi-head attention and the more aggressive multi-query and latent-attention schemes that frontier models now use.
The reason this question separates strong candidates from weak ones is that the arithmetic looks like a triviality and the trap is in the framing. Plenty of people can divide 64 by 8. Far fewer can say precisely why that quotient is also the cache ratio, why the FLOP count does not move, and why the operational payoff shows up as bandwidth on the decode path. Each of those three claims rests on a different fact about how attention and the cache are laid out, so treat them as three small proofs rather than one memorized result.
Counting the KV heads
Grouped-query attention keeps the full set of query heads and partitions them into groups that share a single key head and a single value head. The group size G is how many query heads share one KV head. So the number of KV heads is the number of query heads divided by the group size.
With 64 query heads and group size 8, you split the queries into 8 groups of 8. Each group gets one shared K head and one shared V head. That gives:
So K heads equal V heads equal 8. Crucially, the query head count is untouched. There are still 64 query heads, each computing its own attention. GQA never reduces the queries; it only reduces how many distinct K and V projections the model maintains. Multi-query attention is the extreme case where the group size equals the full head count, leaving exactly one KV head.
It helps to be precise about what "a head" means here. Each query head has its own learned projection matrix that maps the model dimension down to the head dimension. Under MHA, each of those query heads is paired with its own learned K projection and V projection. Under GQA, the K and V projections are simply fewer: the model learns 8 K projections and 8 V projections instead of 64. Query head 0 through 7 all consume the output of K projection 0 and V projection 0. The query projections themselves remain 64 distinct matrices, so the model still asks 64 different questions of the context; it just stores those questions' answers in 8 shared key-value subspaces rather than 64 private ones.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.1 70B ships GQA with 8 KV heads against 64 query heads, the textbook ratio in this question.
- Mistral Large 3 uses GQA to keep its KV cache small enough for long-context serving on a single node.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the reduction factor equal the group size and not the head dim ratio?
Write out the cache formula and isolate which term GQA changes. Only the KV head count moves; head dim and sequence length are held fixed. The ratio of MHA to GQA cache collapses to n_heads over H_kv, which is the group size by definition.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming GQA reduces query heads or adds inference FLOPs. Q heads stay at 64 and arithmetic is unchanged; only the count of cached K and V heads drops.
60 second bullets to scan on the way to the call.
How group size maps query heads to KV heads via a single division
Why the cache reduction factor equals the group size exactly
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.