Llama-2 70B uses 64 query heads, how many KV heads does it actually keep?
Llama-2 70B uses 8 KV heads for 64 query heads, an 8x KV cache reduction via grouped-query attention. The 7B and 13B use plain MHA.
Imagine 64 students all asking different questions in a library. Plain multi-head attention assigns 64 personal librarians and 64 bookshelves, very expensive at scale. Llama-2 70B groups the students into 8 study rooms of 8 each; everyone in one room shares one librarian and one shelf. The questions stay individual (each student keeps their own query), but the reference material is shared. You go from 64 librarians to 8, and the students' results stay nearly as good. That sharing pattern is what GQA does for the KV cache.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Llama-2 70B's choice of 8 KV heads for 64 query heads is one of the cleanest case studies in modern LLM engineering. The decision was driven by inference economics on the H100 generation: at 70B scale with long context, the KV cache dominates HBM consumption, and GQA's 8x reduction is what made interactive long-context serving practical.
The specific number, 8, was picked from a sweep of group sizes, and it sits at the corner of the flat-quality regime in Meta's ablations. This deep dive walks the KV cache math that motivated the choice, why the smaller Llama-2 variants kept plain MHA, the ablation logic behind G = 8 specifically, the operational batch-size implications on H100 hardware, and how the design has evolved through Llama 3, Llama 4 Maverick, and competing model families.
The numbers and what they mean
Llama-2 70B uses 64 query heads and 8 KV heads. Group size G = 64 / 8 = 8. Each KV head is shared across 8 query heads.
Where the numbers come from
Llama-2 70B's config.json exposes the choice via two fields:
num_attention_heads = 64num_key_value_heads = 8
In Hugging Face transformers, that pair of fields defines the GQA configuration. Plain MHA models leave the two equal; MQA models set num_key_value_heads = 1; GQA models pick an intermediate divisor.
What the choice does to attention
Inside the attention block, the 64 query heads compute different attention scores using their own Q projections. The K and V projections, however, produce only 8 head-slices each. The kernel broadcasts each KV head across the 8 query heads in its group, so head 0-7 all consume KV head 0, head 8-15 all consume KV head 1, and so on.
What stays the same
The attention math itself, scaled dot-product attention, is unchanged. The output projection still mixes across all 64 query heads. Positional encoding (RoPE in Llama-2), masking, residual connections, and downstream layers do not need to know that K and V are shared. The GQA change is local to the attention block.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama-2 70B config.json: num_attention_heads = 64, num_key_value_heads = 8. The two-field exposure of GQA in the Hugging Face spec.
- Llama 4 Maverick continues the GQA pattern with similar query-to-KV head ratios at its frontier scale.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did Meta pick GQA-8 rather than MQA for the 70B?
Meta's ablations showed MQA (one KV head) had a measurable perplexity penalty versus MHA across their eval suite. GQA-8 sat on the flat part of the quality curve, with effectively no measured quality loss while still giving an 8x KV cache reduction. The marginal additional saving from going to MQA (8x to 64x) was not worth the quality risk for a flagship 70B release.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming every Llama-2 variant uses GQA. Only the 70B does in the original release. The 7B and 13B kept plain MHA where the KV cache was not the binding constraint.
60 second bullets to scan on the way to the call.
Llama-2 70B query head and KV head counts
Group size G and the relationship num_heads / num_kv_heads
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.