Multi-head attention with N heads of dimension d_head and a single head with dimension d_model = N × d_head have the same total parameter count. Why does multi-head consistently outperform in practice? Reference both the design rationale and interpretability findings.
Same parameters, but multi-head gives N parallel attention distributions per query; a single big head only gives one. Interpretability shows heads specialize.
Imagine you have the same total amount of paint, but you have to choose between one giant brush and a dozen smaller brushes you can use in parallel. The giant brush forces every stroke to do every job at once, outlines, shadows, highlights, and the result is a blurry compromise. The twelve smaller brushes can each focus on one thing. One paints outlines, another paints shadows, another paints highlights. When you layer the strokes together, the finished painting is much richer, even though you used exactly the same amount of paint. Multi-head attention works the same way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-head attention has the same parameter count as a single head attention layer of width d_model. The Q, K, V, and output projection matrices are the same size; multi-head just reshapes them into N concatenated blocks instead of treating them as one. So the consistent quality advantage of multi-head cannot be about capacity. It has to be about what the architecture can express at fixed capacity.
This is the kind of question that's easy to get wrong by reaching for the wrong axis. The intuition 'more heads means more parameters' is just false. The intuition 'more heads means more FLOPs' is also false at matched d_model. The real intuition is structural, and it lives in a property of softmax that is worth being precise about: softmax forces concentration, and one softmax can only concentrate on one place at a time.
We will walk the structural argument, the interpretability evidence, the pruning result that looks like a counterexample but isn't, and what modern compression variants (GQA, MQA, MLA) tell us about which piece of multi-head is actually critical.
The structural argument: softmax concentration
One attention head produces one softmax distribution over positions per query token. That distribution decides which other tokens the query reads from. Softmax pushes mass toward whichever input scores highest, by design. That is what lets attention be selective.
The consequence is that one softmax distribution can emphasize one structural pattern strongly, but it cannot emphasize several different patterns at once. 'Attend to my syntactic head AND attend to the previous token AND attend to my coreferent mention' is three different distributions over three different positions. One softmax cannot be all three simultaneously. Forced to encode all of them at once, it produces a blurry compromise that is worse than any of the three would be individually.
Multi-head fixes this by producing N parallel softmax distributions per token. Each distribution is independent. Each can concentrate on a different relational structure. The outputs of the N heads concatenate, and the next layer sees information drawn from all N attention patterns at once:
The model gets to superpose N different structures per token at the same parameter cost as a single head layer of width d_model. That superposition is the structural win, and it is why multi-head consistently outperforms its single head twin in practice even though the two are parameter-matched and FLOP-matched.
The intuition: one big brush versus N small brushes with the same total paint. The big brush has to do every job at once; the small brushes specialize.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Olsson et al. 2022 mechanistically identified induction heads in GPT-2 and early Anthropic models, pairs of heads driving in context learning.
- Llama 4 Maverick and Mistral Large 3 use many query heads with GQA grouping (8 KV heads), query side diversity preserved, K/V shared.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf specialization is emergent, why doesn't a sufficiently-large single head model recover it through gradient descent?
Because softmax over a single distribution is a hard architectural constraint, not a capacity ceiling. No amount of training can make one probability distribution represent multiple independent structures simultaneously, the model would have to time-share or blur.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Attributing the multi-head win to extra parameters or extra capacity, it's neither; it's parallel attention patterns at the same parameter budget.
60 second bullets to scan on the way to the call.
Parameter count equivalence of single head and multi-head at matched d_model
Why softmax concentration forces one attention pattern per head
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.