**Multi-head attention** is a structural reinterpretation of single head attention, not an additive change.
A quick vocabulary anchor: `d_model` is the model's hidden dimension (768 for BERT-base). A **head** is one parallel attention computation. `d_head` is the per-head feature dimension, defined as:
$$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}$$
In single head attention, the projection weights W_Q, W_K, W_V each have shape (d_model, d_model). The forward pass projects input x to q, k, v of dimension d_model, runs attention, then projects back with W_O of shape (d_model, d_model). Total parameters: 4 × d_model².
In multi-head, the matrices W_Q, W_K, W_V still have shape (d_model, d_model). The forward pass projects to a d_model vector and then **reshapes** that dimension into (num_heads, d_head). Each head runs its own attention over its slice. The outputs are concatenated back to d_model and projected by W_O. Same 4 × d_model² parameters; reshape, not addition.
**False.**
Multi-head attention reshapes the `d_model × d_model` Q/K/V matrices into num_heads parallel d_model × d_head blocks, where d_head = d_model / num_heads. Total parameter count and total FLOPs are unchanged.
The way I'd think about it: the underlying matrix is the same size. You're just choosing whether to interpret the output dimension as one chunk of size d_model or as num_heads parallel chunks of size d_head. Storage is identical, and the matmul work decomposes into smaller per-head matmuls that sum to the same total FLOPs.
What does change is structural: you get num_heads parallel attention distributions per query instead of one, which enables head specialization. That's the actual reason multi-head exists, same parameter budget, more expressive routing.
Quick reference
TL;DR
False. Multi-head is reshape, not parameter add. The Q/K/V matrices stay d_model × d_model: just sliced into num_heads × d_head blocks.
Think of a shelf with a fixed total width holding books. One head is one giant book covering the whole shelf. Twelve heads is the same shelf width split into twelve narrower book slots side by side. The wood you used to build the shelf, the total parameters, is exactly the same. You only changed how the shelf is divided. Same lumber, different compartments, twelve mini-readers instead of one big one.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very small models (<100M parameters) at small d_model At `d_model = 256`, even `d_head = 64` means only 4 heads. Slicing further hits the floor where each head loses representational capacity, single head or 2 heads often matches quality at lower kernel overhead.
Very large models where d_model has grown past 8192 Holding `d_head = 64` means `n_heads = 128+`, which inflates KV cache and kernel launch overhead. Modern frontier models scale `d_head` to 128 (or use GQA / MLA) to keep head count manageable.
Inference latency critical decode at very long context Each extra K/V head costs memory bandwidth in the KV-cache read on every decode step. Switching to GQA (fewer K/V heads at same query head count) cuts that bandwidth without losing the parallel distribution benefit.
Edge / mobile deployment with tight FLOP budgets Smaller `d_head` doesn't reduce total FLOPs at fixed `d_model`, but fewer heads with larger `d_head` can map better onto matrix engines that don't have efficient kernels at `d_head < 64`.
Quick scan · 2–4 min
Everything important, quickly.
Walk through the reshape mechanics, confirm parameter and FLOP equivalence, contrast with GQA/MQA which DO save K/V parameters, and explain why the structural difference (parallel attention patterns) is the real reason multi-head exists.
Side-by-side
Configuration
d_model
num_heads
d_head
Total params (Q+K+V+O)
single head
768
1
768
4 × 768²
multi-head (12)
768
12
64
4 × 768²
multi-head (8)
768
8
96
4 × 768²
bigger model
1024
12
85.3 (invalid)
4 × 1024²
Real-world examples
Real products, models, and research that use this idea.
1Switching BERT-base from 12 to 6 heads at fixed d_model=768 changes d_head from 64 to 128, same total params, different structural budget.
2Llama 3 8B vs Llama 3 70B differ in d_model (4096 vs 8192) not just num_heads: that's where the parameter scaling comes from.
3GQA in Llama 3 reduces K/V heads from 64 to 8, this DOES save parameters on W_K and W_V (since they shrink), unlike standard multi-head reshape.
4MQA in PaLM uses 1 shared K/V head for all query heads: substantial parameter savings on W_K and W_V.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the parameter difference between standard MHA, GQA, and MQA?
Sign in or go Pro to see the approach for this follow-up.
QIf multi-head is parameter-neutral, why isn't d_head usually chosen very small (like 8 or 16)?
Sign in or go Pro to see the approach for this follow-up.
QDoes the reshape add any compute overhead?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming each head 'has its own' projection matrices that get added on top, they're slices of the same matrix, not additions.