If d_model = 768 and num_heads = 12, what is d_head?

**Multi-head attention** takes a `d_model`-dimensional input and splits its representational space evenly across `num_heads` parallel attention heads. The size each head operates on is called `d_head`, and the rule is: $$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}$$ For BERT-base, the most-cited reference config: d_model = 768 and num_heads = 12. So d_head = 768 / 12 = 64. Mechanically: the Q, K, V projections produce tensors of shape `(batch, seq, d_model)`. To run multi-head attention you reshape that d_model axis into (num_heads, d_head), for BERT-base, that's reshaping a 768-dim vector into 12 chunks of 64 dimensions. Each chunk is processed by one head in parallel, and the heads are concatenated and projected back at the end. The constraint is that d_model must be evenly divisible by num_heads, otherwise the reshape fails. That's why typical num_heads values are clean divisors of d_model (8, 12, 16). The 64-per-head convention is everywhere because it's a sweet spot, big enough to give each head meaningful capacity, small enough to keep per-head matmuls efficient on GPU tensor cores.

768 divided by 12 is 64, that's d_head for BERT-base, straight from the partition rule: $$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}} = \frac{768}{12} = 64$$ The rule is just integer division, `d_model` has to be evenly divisible by `num_heads`, otherwise the reshape from `(B, T, d_model)` to `(B, T, n_heads, d_head)` errors at runtime. What's worth flagging in an interview is that `d_head = 64` isn't an accident. It's one of two canonical values (the other being 128 for larger models). The convention is hardware driven: NVIDIA tensor cores want multiples of 8 or 16 in matrix dimensions, and FlashAttention kernels are heavily tuned for `d_head ∈ {64, 128}`. Pick something weird like 80 and you fall off the fast path. The interesting bit is that `d_head` has stayed at 64-128 across model scales while `d_model` has grown massively, Llama-3 70B uses `d_model = 8192` with `d_head = 128`. The implicit hypothesis: per-head capacity is bounded by what's useful for ONE attention pattern, so you scale by adding more heads, not by making each head bigger.