What does multi-head attention give you that a single attention head with the same total parameters cannot?

MCQ·Medium·4.0 · 0·~1 min·Relevant atMicrosoft

Attempt it

Multi-head attention is the workhorse pattern of every modern transformer. To see why it beats a single attention head with the same parameter budget, you have to understand what 'attention pattern' actually means. Attention from one token over a sequence produces a probability distribution over the other tokens: a vector of weights that sums to one. The token's output is then a weighted sum of value vectors using those weights. In a single head setup there is exactly one such distribution per query token per layer. The model has to use it for everything: syntactic dependencies, positional information, lexical matching, coreference resolution. These are different kinds of relationships that want different attention weights, and a single distribution has to pick one or produce a blurry compromise. Multi-head sidesteps this by running N attention computations in parallel. The key bit is that it does not add parameters, it reshapes them. The same projection matrices that a single head version would use are reinterpreted as N parallel blocks of smaller width. Each block computes its own attention and produces its own small output vector. The N outputs are concatenated and projected back to the original width. The consequence: each head gets its own attention distribution. One head can specialize in 'attend to the previous token', another in 'attend to syntactic subjects', another in 'attend to the most recent mention of this entity'. Interpretability research has shown this specialization emerges from training: nobody programs it in. The single head model cannot do this because it only has one distribution to spend.

Yeah, the way I think about this is, multi-head gives you N parallel attention distributions per layer where single head gives you exactly one, and the structural advantage is having those parallel distributions, not extra capacity. A single attention head produces one probability distribution over positions per query token. That single distribution has to encode every kind of relationship the model cares about at that layer: syntax, position, coreference, lexical matching. It cannot do all of them well at the same time. Multi-head reshapes the same projection matrices into N parallel blocks of width d-model over N. Each block runs its own attention and produces its own distribution. The outputs concatenate and pass through a final output projection: $$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$$ Parameter count and FLOPs are essentially unchanged. The win is structural: N parallel distributions instead of one. The trade off worth naming is that the specialization is emergent, not designed. Interpretability work, Voita, Clark, the induction head literature, has documented heads that visibly specialize in syntactic dependencies, positional patterns, or in context induction. You get that diversity for free, but only because the architecture makes it possible.

Quick reference

TL;DR

Multi-head produces N parallel attention distributions per layer instead of one, letting different heads specialize in syntax, position, and coreference at the same parameter and FLOP budget.

Memory aid

Easy to grasp

Imagine reading a sentence with only one highlighter, you can mark one kind of relationship at a time, say which word goes with which verb, and you have to choose what to track. Multi-head attention is like having twelve different-coloured highlighters and using them all at once: one tracks verbs, one tracks pronouns, one tracks adjacent words. You see all the structures at the same time instead of being forced to pick one. The total amount of ink is the same as one big highlighter; you just spread it across colours.

Key concepts

Multi-head attention Mechanistic interpretability

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: reshape not parameter add; one distribution versus N distributions; emergent specialization (induction, syntactic, positional heads); prunability and training time diversity; modern variants MQA/GQA/MLA preserving query side diversity.

Code example

python

# Multi-head attention is a reshape, not extra parameters.
# d_model = n_heads * d_head
import math
import torch
import torch.nn.functional as F

B, T, d_model, n_heads = 2, 16, 512, 8
d_head = d_model // n_heads

x = torch.randn(B, T, d_model)
W_q = torch.nn.Linear(d_model, d_model, bias=False)
W_k = torch.nn.Linear(d_model, d_model, bias=False)
W_v = torch.nn.Linear(d_model, d_model, bias=False)
W_o = torch.nn.Linear(d_model, d_model, bias=False)

q = W_q(x).view(B, T, n_heads, d_head).transpose(1, 2)
k = W_k(x).view(B, T, n_heads, d_head).transpose(1, 2)
v = W_v(x).view(B, T, n_heads, d_head).transpose(1, 2)

scores = (q @ k.transpose(-2, -1)) / math.sqrt(d_head)
attn = F.softmax(scores, dim=-1)
out = (attn @ v).transpose(1, 2).reshape(B, T, d_model)
out = W_o(out)

Real-world examples

Real products, models, and research that use this idea.

GPT-5.5 and Claude Opus 4.7 both ship deep transformer stacks with many parallel heads per layer, combined with grouped-query attention to shrink the KV cache at long context.
Llama 4 Maverick uses many query heads per layer with a small number of grouped K/V heads: query side diversity preserved, K/V side compressed for inference.
DeepSeek V4 ships multi-head latent attention, which keeps multiple query heads while compressing the K/V cache into a smaller latent space.
Olsson et al. 2022 'In-context Learning and Induction Heads' is the canonical example of emergent head specialization driving in context learning behavior.
The original Vaswani et al. 2017 transformer used 8 heads with d-model 512 and head dim 64.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf many heads are prunable post-training, why train with so many in the first place?

QHow do MQA, GQA, and MLA preserve the multi-head win while saving inference memory?

QCould you replace multi-head with a single head plus a mixture of experts router?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Many candidates claim multi-head adds parameters or reduces compute. It does neither, it is a reshape that buys parallel attention patterns at constant cost.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.