Why does multi-head attention outperform a single large attention head?

Start with the parameter count equivalence. A single attention head with dimension `d_model` has Q, K, V projections of size `d_model × d_model`. **Multi-head** with N heads of dim `d_head = d_model / N` has the same total parameter count, just interpret the Q, K, V matrices as N concatenated `(d_model, d_head)` blocks. So where does the quality gap come from? It is structural. $$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$$ An attention head produces one probability distribution per query: one weighting over positions that decides which tokens to read from. Softmax forces concentration, so that distribution cannot be 'a little syntax + a little position + a little coreference' at once. One head has to pick one structure or blur all of them. Multi-head escapes this by producing N parallel distributions, each free to concentrate on a different relationship type. Outputs concatenate; the next layer sees all N patterns at once. Interpretability confirms this is what happens. Voita 2019, Clark 2019, and Olsson 2022 found heads visibly specialize (syntactic, positional, induction, coreference roles) all emergent from training. Michel 2019 showed many heads are prunable post-training, but training needs the diversity to find the specializations.

Same parameter count, but different shape. A single attention head produces one probability distribution per query. Softmax forces that distribution to concentrate, so it can encode at most one structural relationship at a time. Multi-head reshapes the same projection budget into N parallel distributions whose outputs concatenate: $$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$$ The model can now attend to N different relationship types simultaneously, syntactic, positional, coreference, at the same total parameter cost. Interpretability work confirms this is what actually happens. Voita 2019 and Olsson 2022 found that trained heads visibly specialize, and the specializations emerge from gradient descent without any explicit supervision. Head-pruning research shows post-hoc redundancy among trained heads, but that's a separate observation. Training still needs the diversity to find the useful specializations in the first place.