Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
**Grouped-query attention (GQA)** is a middle ground between full multi-head attention (every query head has its own K and V) and multi-query attention (all query heads share a single K and V).
In GQA you pick `n_kv_groups` smaller than `n_heads`. The query heads are split into that many equal groups, and each group shares one K and one V projection. This shrinks the KV cache without sacrificing as much quality as MQA.
The configuration above asks for 32 query heads and 7 KV heads. That fails because 32 is not divisible by 7. Each KV head must serve the same number of query heads (the group size must be a whole number), so the only legal choices for `n_kv_heads` are divisors of 32: 1, 2, 4, 8, 16, or 32. Llama-2 70B uses 8, which is almost certainly what the teammate meant.
The model does not silently truncate or pad. The attention reshape from `(batch, seq, 32, d_head)` into groups raises a shape mismatch the first time it executes, well before anything reaches staging.
**GQA** needs `num_attention_heads % num_key_value_heads == 0`. The reshape from a flat 32-head tensor into `(n_kv_groups, group_size, d_head)` only works when 32 factors cleanly into the product. 32 mod 7 = 4, so the configuration is broken.
The error does not surface in the smoke test the way you might expect. The config file parses fine because both numbers are positive integers. The weights load fine because the K and V projection matrices have shape `(d_model, 7 * d_head)`, which is dimensionally legal on its own. The crash comes at the first forward pass, when the attention kernel tries to broadcast 7 KV heads across 32 query heads and the reshape rejects the non-integer group size.
Valid choices for 32 query heads are the divisors: 1 (pure MQA), 2, 4, 8, 16, or 32 (pure MHA). Llama-2 70B settled on 8 as the production sweet spot. The fix is almost certainly to change 7 to 8 in the config.
The broader lesson is that 'the model file loaded' is not a meaningful smoke test for shape constraints. A unit test that runs a single forward pass on a batch of two tokens catches GQA misconfigurations in under a second.
Quick reference
TL;DR
GQA requires num_attention_heads % num_key_value_heads == 0. 32 mod 7 = 4, so the reshape fails at load time and 7 was almost certainly a typo for 8.
Picture 32 students in a classroom that need to be split into equal teams sharing one shared textbook per team. If you ask for 8 teams, every team gets 4 students; clean and fair. If you ask for 7 teams, the math fails: you cannot hand the same 32 students to 7 teams of equal size, because 32 does not divide cleanly into 7 parts. GQA works the same way. The 32 query heads are the students, the KV heads are the shared textbooks, and each KV head must serve the same number of query heads. Pick a divisor of 32 (1, 2, 4, 8, 16, 32) and life is easy. Pick 7 and the reshape blows up the moment the model tries to run.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Architectures using multi-latent attention (MLA) like DeepSeek V4 where KV is shared via low-rank compression MLA replaces per-head KV with a shared low-rank latent, so the head-count divisibility constraint does not apply in the same form.
Pure multi-head attention (MHA) where every query head has its own K and V MHA sets n_kv_heads == n_heads, so divisibility is trivially satisfied and the constraint is irrelevant.
Pure multi-query attention (MQA) where all heads share one K and one V MQA sets n_kv_heads == 1, which divides any n_heads, so the constraint is again trivially satisfied.
Research prototypes that intentionally implement custom kernels with scatter or pad logic A bespoke kernel can support unequal group sizes, but every mainstream production stack assumes the divisibility constraint.
Quick scan · 2–4 min
Everything important, quickly.
Walk through the GQA reshape mechanics, explain why divisibility is a kernel constraint not an algorithmic one, identify where the error surfaces (forward pass, not config load), name 8 as the likely intended value, and end with a CI unit test that prevents future repeats.
Code example
python
# Validation that should live in your model config loader.
def validate_gqa_config(n_heads: int, n_kv_heads: int) -> None:
if n_heads % n_kv_heads != 0:
raise ValueError(
f"GQA requires n_heads ({n_heads}) divisible by "
f"n_kv_heads ({n_kv_heads}). "
f"For {n_heads} query heads, legal n_kv_heads values are "
f"divisors of {n_heads}: "
f"{[d for d in range(1, n_heads + 1) if n_heads % d == 0]}"
)
# This catches the bug at PR review, not at the first forward pass.
validate_gqa_config(n_heads=32, n_kv_heads=7) # raises ValueError
validate_gqa_config(n_heads=32, n_kv_heads=8) # passes (group_size=4)
Real-world examples
Real products, models, and research that use this idea.
3Mixtral 8x7B: 32 query heads, 8 KV heads per expert, group size 4.
4DeepSeek V4 (2025): uses MLA, a generalization of GQA that compresses KV via a low-rank latent rather than via head sharing.
5Qwen 3.5 (2026): family of GQA configurations across model sizes, all preserving the head divisibility constraint.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did Llama-2 70B pick exactly 8 KV heads for 64 query heads rather than 4 or 16?
Sign in or go Pro to see the approach for this follow-up.
QHow does MLA in DeepSeek V4 sidestep the head-divisibility constraint entirely?
Sign in or go Pro to see the approach for this follow-up.
QIf your team needed 7-way KV grouping for some hardware reason, how would you actually implement it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the config loaded successfully because the model file opened, missing that the shape error only fires when the attention reshape executes during the first real forward pass.