**Multi-head attention** takes a `d_model`-dimensional input and splits its representational space evenly across `num_heads` parallel attention heads. The size each head operates on is called `d_head`, and the rule is:
$$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}$$
For BERT-base, the most-cited reference config: d_model = 768 and num_heads = 12. So d_head = 768 / 12 = 64.
Mechanically: the Q, K, V projections produce tensors of shape `(batch, seq, d_model)`. To run multi-head attention you reshape that d_model axis into (num_heads, d_head), for BERT-base, that's reshaping a 768-dim vector into 12 chunks of 64 dimensions. Each chunk is processed by one head in parallel, and the heads are concatenated and projected back at the end.
The constraint is that d_model must be evenly divisible by num_heads, otherwise the reshape fails. That's why typical num_heads values are clean divisors of d_model (8, 12, 16). The 64-per-head convention is everywhere because it's a sweet spot, big enough to give each head meaningful capacity, small enough to keep per-head matmuls efficient on GPU tensor cores.
768 divided by 12 is 64, that's d_head for BERT-base, straight from the partition rule:
$$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}} = \frac{768}{12} = 64$$
The rule is just integer division, `d_model` has to be evenly divisible by `num_heads`, otherwise the reshape from `(B, T, d_model)` to `(B, T, n_heads, d_head)` errors at runtime.
What's worth flagging in an interview is that `d_head = 64` isn't an accident. It's one of two canonical values (the other being 128 for larger models). The convention is hardware driven: NVIDIA tensor cores want multiples of 8 or 16 in matrix dimensions, and FlashAttention kernels are heavily tuned for `d_head ∈ {64, 128}`. Pick something weird like 80 and you fall off the fast path.
The interesting bit is that `d_head` has stayed at 64-128 across model scales while `d_model` has grown massively, Llama-3 70B uses `d_model = 8192` with `d_head = 128`. The implicit hypothesis: per-head capacity is bounded by what's useful for ONE attention pattern, so you scale by adding more heads, not by making each head bigger.
Quick reference
TL;DR
d_head = d_model / num_heads = 768 / 12 = 64. Memorize the BERT-base config; it appears constantly.
Imagine a long shelf with 768 books on it, and you want 12 librarians to each look after their own section. You'd give each librarian 768 / 12 = 64 books to manage. In multi-head attention, the shelf is the model dimension, the librarians are the attention heads, and each head's 64-book section is what we call d_head.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking d_head < 32 to fit more heads into a small d_model Per-head capacity collapses below ~32 dims, heads can't represent enough distinct attention patterns and the variance argument behind √d_k stops giving meaningful normalization. Quality degrades faster than the extra heads compensate.
Choosing d_head ∉ {64, 128} on GPU-served inference FlashAttention v2 / v3 kernels are heavily tuned for d_head = 64 and 128. Values like 80, 96, or 192 fall to generic paths and lose 20-40% wall-clock throughput even when arithmetic is identical.
Scaling d_head linearly with d_model on very large models Empirically d_head plateaus at 128-192 across the largest production LLMs. Spending parameters on bigger heads is dominated by spending them on more heads or more layers.
d_model not divisible by num_heads in any planned config The reshape (B, T, d_model) → (B, n_heads, T, d_head) errors at runtime. Either pad d_model up to the nearest multiple or pick a num_heads that divides cleanly: there is no soft fail.
Quick scan · 2–4 min
Everything important, quickly.
State the formula, give the BERT-base number, explain divisibility constraint, name common configs (BERT, GPT, Llama), and mention hardware convention of d_head ∈ {64, 128} for FlashAttention compatibility.
Side-by-side
Model
d_model
num_heads
d_head
BERT-base
768
12
64
BERT-large
1024
16
64
GPT-2 small
768
12
64
GPT-3 175B
12288
96
128
Llama 3 70B
8192
64
128
Real-world examples
Real products, models, and research that use this idea.