True or false: switching from 1-head to 12-head attention at the same d_model increases total parameter count.
False. Multi-head is reshape, not parameter add. The Q/K/V matrices stay d_model × d_model: just sliced into num_heads × d_head blocks.
Think of a shelf with a fixed total width holding books. One head is one giant book covering the whole shelf. Twelve heads is the same shelf width split into twelve narrower book slots side by side. The wood you used to build the shelf, the total parameters, is exactly the same. You only changed how the shelf is divided. Same lumber, different compartments, twelve mini-readers instead of one big one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The 1-head vs 12-head question at fixed d_model is one of the cleanest tests of whether a candidate has internalized the arithmetic of attention or just memorized the slogan 'multi-head is better'.
It feels like a trap because option D (no difference at all) is obviously wrong, so 'False with no difference' candidates flip to 'True, more parameters'. Both poles are wrong. The truth is structural: parameter-neutral, FLOP neutral, but architecturally distinct.
The whole question is whether you can compute
n_heads × d_headand notice it equalsd_model.
The reshape mechanic in code
Walk the actual tensor shapes through a forward pass. Suppose B = 1, T = 512, d_model = 768, n_heads = 12, d_head = 64.
x: (1, 512, 768)
q = x @ W_Q # W_Q is (768, 768), output (1, 512, 768)
q = q.view(1, 512, 12, 64).transpose(1, 2) # → (1, 12, 512, 64)
At this point we have 12 logical heads, each operating on a 64-dim subspace, but the underlying memory is still the single 768-dim projection's output. The view and transpose are zero-FLOP layout operations.
The attention math then runs per-head:
- QK^T: (1, 12, 512, 64) @ (1, 12, 64, 512) → (1, 12, 512, 512)
- softmax along the last dim, optionally with a causal mask
- weights @ V: (1, 12, 512, 512) @ (1, 12, 512, 64) → (1, 12, 512, 64)
Finally:
out = out.transpose(1, 2).contiguous().view(1, 512, 768)
out = out @ W_O # W_O is (768, 768)
Notice every weight matrix is (768, 768). There are exactly four of them: W_Q, W_K, W_V, W_O. Total: 4 × 768² ≈ 2.36M parameters per attention block. Independent of head count.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Configuration | d_model | num_heads | d_head | Total params (Q+K+V+O) |
|---|---|---|---|---|
| single head | 768 | 1 | 768 | 4 × 768² |
| multi-head (12) | 768 | 12 | 64 | 4 × 768² |
| multi-head (8) | 768 | 8 | 96 | 4 × 768² |
| bigger model | 1024 | 12 | 85.3 (invalid) | 4 × 1024² |
Real products, models, and research that use this idea.
- Switching BERT-base from 12 to 6 heads at fixed d_model=768 changes d_head from 64 to 128, same total params, different structural budget.
- Llama 3 8B vs Llama 3 70B differ in d_model (4096 vs 8192) not just num_heads: that's where the parameter scaling comes from.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the parameter difference between standard MHA, GQA, and MQA?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming each head 'has its own' projection matrices that get added on top, they're slices of the same matrix, not additions.
60 second bullets to scan on the way to the call.
The invariant n_heads × d_head = d_model
Why the Q/K/V projection matrices are (d_model × d_model) independent of head count
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.