**False.** Swapping a single-head attention layer for a 12-head one with the same d_model does NOT change the parameter count. The Q, K, V projection matrices are `d_model x d_model` in both cases.
In single-head attention, you have one Q, one K, one V projection, each a `d_model x d_model` matrix. Total parameters in the projection layers: `4 x d_model^2` (counting the output projection W_O).
In multi-head attention with h heads, you have... the same matrices. They are just viewed differently. A `d_model x d_model` projection can be reshaped into `d_model x h x d_head` where `d_head = d_model / h`. The reshape is a view, not new parameters.
What changes is HOW you use those parameters at compute time. The single big matrix is split into h smaller blocks. Each block runs its own scaled dot-product attention in parallel. The outputs are concatenated back together and passed through W_O.
The whole point of multi-head attention is parallel specialization at the same parameter cost. You do not pay extra in weights; you gain expressiveness by letting h heads attend to different relations inside the same parameter budget.
**False.** The Q, K, V, and O projections are each `d_model x d_model` regardless of head count. Multi-head attention is a reshape of these big matrices into `h` smaller blocks of width `d_head = d_model / h`.
Specifically, for a single attention layer:
- W_Q: shape `(d_model, d_model)`, viewed as `(d_model, h, d_head)`.
- W_K: shape `(d_model, d_model)`, viewed as `(d_model, h, d_head)`.
- W_V: shape `(d_model, d_model)`, viewed as `(d_model, h, d_head)`.
- W_O: shape `(d_model, d_model)`, viewed as `(h * d_head, d_model)`.
Total parameter count: `4 * d_model^2`, independent of h. Going from h=1 to h=12 changes nothing in the weight count.
What the multi-head structure buys is parallel specialization. Each head sees a smaller `d_head`-wide subspace and learns to attend to a specific kind of relation: positional adjacency, syntactic structure, coreference, induction. Probing studies on BERT and GPT have repeatedly found this specialization without any explicit constraint at training time.
The practical insight: if you want more attention capacity, increase d_model (which quadratically grows the projection parameters) or add more layers. Splitting an existing d_model across more heads only reshapes the existing budget, it does not add capacity.
Quick reference
TL;DR
False. Multi-head attention reshapes the existing d_model-wide projections into h smaller blocks; the parameter count is identical to single-head at the same d_model.
Picture a giant table you can either use as one large workspace or partition with movable dividers into 12 smaller workspaces. The wood is the same; the dividers are free. Multi-head attention is the partitioned version of the same table, no extra material spent. What you gain is the ability for 12 people to work in parallel on different problems, not a bigger workspace.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
When you actually need more parameter capacity Adding heads at fixed d_model does not add capacity; you need a larger d_model or more layers.
When d_model is small enough that d_head becomes too narrow Each head's subspace must be wide enough to encode useful relations; very high head counts at small d_model can collapse per-head expressiveness.
When KV cache memory is the binding constraint More heads under MHA inflates KV cache linearly; the parameter count argument is irrelevant if memory is the bottleneck.
When comparing across architectures with different per-head structures GQA, MQA, and MLA change the K, V structure so direct parameter counting differs from the MHA case.
Quick scan · 2–4 min
Everything important, quickly.
4m: the projection shape accounting, why multi-head is a reshape and not new parameters, the parallel-specialization inductive-bias argument, the distinction between parameter count and capacity, and the KV cache asymmetry.
Real-world examples
Real products, models, and research that use this idea.
1BERT-base (h=12, d_model=768) and a hypothetical single-head BERT-base (h=1, d_model=768) would have identical parameter counts.
2Llama-3 70B has 64 heads at d_model=8192; collapsing to h=1 at the same d_model keeps the same projection parameters.
3Probing studies on BERT and GPT consistently find head specialization (induction heads, coreference heads) at fixed parameter counts.
4When papers report 'increasing num_heads' as an ablation, they usually keep d_model fixed and show the parameter count line is flat.
5GQA changes num_kv_heads independently of num_heads, decoupling cache size from query-side parameter structure.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf multi-head adds no parameters, why does it consistently outperform single-head at the same d_model?
Sign in or go Pro to see the approach for this follow-up.
QHow does num_heads affect KV cache size even though it does not affect parameter count?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right way to add real capacity to an attention layer?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming each head adds its own full-width (d_model x d_head) projection on top of the existing one. The heads slice the existing matrix; they do not add to it.