d_head = d_model / num_heads = 768 / 12 = 64. Memorize the BERT-base config; it appears constantly.
Imagine a long shelf with 768 books on it, and you want 12 librarians to each look after their own section. You'd give each librarian 768 / 12 = 64 books to manage. In multi-head attention, the shelf is the model dimension, the librarians are the attention heads, and each head's 64-book section is what we call d_head.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
d_head is the per-head dimension of multi-head attention, defined by the integer division d_head = d_model / num_heads. For BERT-base (d_model = 768, num_heads = 12), d_head = 64, a number every transformer engineer eventually memorizes.
The arithmetic is trivial. What makes the question interesting is why this specific value, and why it has barely budged across six years of frontier LLMs that grew d_model by more than 10×.
This deep dive covers the mechanical role of d_head in the multi-head reshape, the variance derivation that ties d_head to the √d_k scaling, the hardware convention that pins d_head to 64 or 128, and the modern variants (GQA, MQA, MLA) that keep the query side d_head intact while compressing K and V.
Where d_head lives in the multi-head reshape
Multi-head attention runs num_heads parallel attention computations on disjoint slices of the representation, then concatenates the outputs and applies a final linear projection.
The reshape pipeline
Starting from input x : (B, T, d_model):
- Project to Q, K, V, each (B, T, d_model) via W_Q, W_K, W_V : d_model × d_model.
- Reshape d_model → (n_heads, d_head), giving (B, T, n_heads, d_head).
- Transpose to (B, n_heads, T, d_head) so each head is a leading batch dim.
- Run scaled dot-product attention per head in parallel.
- Transpose back, reshape (n_heads, d_head) → d_model, project with W_O.
The divisibility constraint
The reshape in step 2 requires d_model % num_heads == 0. There is no soft fail, the tensor reshape raises an error at runtime if the constraint is violated.
Practical rule: pick num_heads to be a clean divisor of d_model, or pad d_model up. Don't pick num_heads first and hope the math works out.
d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Model | d_model | num_heads | d_head |
|---|---|---|---|
| BERT-base | 768 | 12 | 64 |
| BERT-large | 1024 | 16 | 64 |
| GPT-2 small | 768 | 12 | 64 |
| GPT-3 175B | 12288 | 96 | 128 |
| Llama 3 70B | 8192 | 64 | 128 |
Real products, models, and research that use this idea.
- BERT-base: d_model=768, num_heads=12, d_head=64.
- BERT-large: d_model=1024, num_heads=16, d_head=64.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy has d_head stayed at 64-128 across model scales while d_model has grown massively?
Per-head capacity is bounded by what's useful for ONE attention pattern. Beyond ~128 dims per head, returns diminish because each head still only emits one attention distribution. Scaling model capacity is better done by adding more heads or more layers.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that d_head must divide d_model evenly, pick num_heads that divides d_model cleanly, or change d_model.
60 second bullets to scan on the way to the call.
How d_head, d_model, and num_heads relate
BERT-base config (768, 12, 64) from memory
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.