Multi-head attention splits the work across `num_heads` parallel heads. The split looks more complicated than it is, it is really just a reshape.
Start from a token embedding `x` of shape `(d_model,)`. Three projections produce Q, K, and V:
1. `Q = x @ W_Q` → shape `(d_model,)`
2. `K = x @ W_K` → shape `(d_model,)`
3. `V = x @ W_V` → shape `(d_model,)`
Now Q, K, V are still d_model wide. The 'split into heads' step is a **view + transpose**:
- Reshape Q from `(d_model,)` to `(num_heads, d_head)` where `d_head = d_model / num_heads`.
- No memory copy; PyTorch reinterprets contiguous chunks of the same buffer as a new axis.
Each head then performs scaled dot-product attention on its own `(d_head,)`-wide slice. At the end, the per-head outputs are concatenated back into a `(d_model,)` vector and run through the output projection `W_O`.
So the answer to 'how is the embedding split' is: through one big projection followed by a reshape that re-labels contiguous chunks as heads.
Multi-head attention does **not** duplicate the embedding per head. It uses a single batched projection followed by a reshape.
Given input `x` of shape `(batch, seq, d_model)`, the pipeline is:
1. `Q = x @ W_Q`, same for K and V. Each `W_*` is `(d_model, d_model)`. Output stays `(batch, seq, d_model)`.
2. Reshape Q, K, V to `(batch, seq, num_heads, d_head)` where `d_head = d_model / num_heads`. This is a **view**, not a copy.
3. Transpose to `(batch, num_heads, seq, d_head)` so each head's tensor is contiguous along the sequence axis.
4. Run scaled dot-product attention independently per head: `softmax(Q_h K_h^T / sqrt(d_head)) V_h` for each head h.
5. Concatenate the heads back along the last axis into `(batch, seq, d_model)`.
6. Apply the output projection `W_O` of shape `(d_model, d_model)`.
The critical insight: the d_model-wide Q projection IS the per-head Q projections concatenated. The model learns one big matrix; the reshape just exposes the head structure. There is no copy, no duplication, no extra parameters compared to a hypothetical 'one projection per head' design.
Quick reference
TL;DR
After the Q, K, V projections produce d_model-wide vectors, a reshape + transpose splits each vector into (num_heads, d_head), a tensor view, not a copy.
Picture a long candy bar. You can cut it into 8 equal pieces or call it one bar with 8 labeled sections; the candy is the same either way. Multi-head attention does the second thing. The token vector is one long buffer in memory, and the model just relabels contiguous chunks of it as 'this part is head 1, this part is head 2', and so on. Nothing is copied or duplicated. Each head then does its own small attention computation on its slice, and the model concatenates the slices back together at the end.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-head attention models or very small toy models Reshape overhead is not zero for ultra-small tensors and the multi-head split adds code complexity without quality wins below d_model = 256 or so.
Mixture of experts attention designs with heterogeneous head shapes Some MoE variants use non-uniform head dimensions across experts; the clean (num_heads, d_head) reshape pattern breaks and a more general routing layer is needed.
Quantization-aware training with mixed-precision per-head scales If each head needs its own quantization scale, you may want explicit per-head tensors instead of a view, so the scales can be applied independently before the matmul.
Quick scan · 2–4 min
Everything important, quickly.
Trace the shape transformations from x through QKV projection, reshape, transpose, per-head attention, concat, and W_O. Highlight that the split is a view, name the GQA/MQA variations, and end with why per-head softmax is the real reason multi-head wins.
Real-world examples
Real products, models, and research that use this idea.
1PyTorch's `nn.MultiheadAttention` uses exactly this view and transpose pattern internally.
2Llama 4 Maverick uses grouped-query attention: Q stays full multi-head while K and V share groups, but the reshape mechanism is identical.
3FlashAttention-3 expects the (batch, num_heads, seq, d_head) layout after reshape; the kernel's block streaming depends on it.
4Mistral 7B and Llama 4 typically use d_model = 4096 and num_heads = 32, giving d_head = 128,the most common modern configuration.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf multi-head attention is mathematically equivalent to one big head with the same total dimension, why use multiple heads at all?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes structurally when you go from MHA to GQA to MQA in the projection and reshape?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing each head has its own private d_model-wide projection. The projections produce one d_model vector that is then sliced into per-head pieces.