Llama-2 7B and Mistral 7B both use d_model = 4096 with 32 query heads, giving d_head = 128. Smaller 1-3B models typically use around 16 heads.
Picture splitting a wide highway into lanes. The highway has 4096 lanes total (that is d_model). Llama-2 7B and Mistral 7B both cut it into 32 groups of lanes, so each group is 128 lanes wide. That count, 32, is the number of attention heads. Smaller cars (1-3B models) make do with about 16 groups of lanes because they have less traffic to route. The arithmetic is just d_model = num_heads x d_head. The split 4096 = 32 x 128 is the standard that lots of open-weight LLMs settled on.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The 7B-class LLM configuration converged on a specific set of numbers between 2023 and 2024: d_model = 4096, num_heads = 32, d_head = 128. Both Llama-2 7B and Mistral 7B use this exact configuration, and most later 7B open-weight LLMs followed. The convergence is not coincidence. It reflects the d_model = num_heads x d_head identity that most models follow, kernel efficiency (FlashAttention is most tuned for d_head = 64 or 128), and scaling-law guidance (d_model around 4096 for the 7B parameter target).
The constraint: d_model = num_heads x d_head
Multi-head attention reshapes a flat d_model vector into a 2D (num_heads, d_head) block before the per-head matmul. In a standard implementation the reshape needs d_model = num_heads * d_head, with strict integer divisibility.
Why the identity holds
The Q, K, V projections produce tensors of shape (batch, seq_len, d_model). To split into heads, the implementation views this as (batch, seq_len, num_heads, d_head). The view operation needs the trailing dimensions to multiply back to d_model. If they do not, PyTorch (or any framework) errors out at the reshape.
What this means for choosing numbers
You get two degrees of freedom, not three. Pick d_model and num_heads, and d_head follows. Pick d_model and d_head, and num_heads follows. Pick num_heads and d_head together, and you have fixed d_model.
The convention in modern LLMs is to pick d_model first (based on parameter-count targets and scaling laws). Then pick d_head from a small kernel-friendly set ({64, 128}), and let num_heads emerge.
A worked example
For a 7B target with d_model = 4096 and d_head = 128:
num_heads = d_model / d_head = 4096 / 128 = 32
For a 70B target with d_model = 8192 and d_head = 128:
num_heads = 8192 / 128 = 64
The head dim stays constant; everything else scales.
In a standard implementation, picking incompatible values produces a runtime error. The only way around it is to give the Q, K, V projections an output width of num_heads x d_head that differs from d_model. Gemma 7B does this: d_model = 3072, with 16 heads x 256 = 4096 query values per token.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama-2 7B config: hidden_size = 4096, num_attention_heads = 32, head_dim = 128.
- Mistral 7B config: hidden_size = 4096, num_attention_heads = 32, head_dim = 128 (identical to Llama-2 7B).
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy has d_head = 128 become so standard across model sizes?
Kernel efficiency dominates. FlashAttention v2 and v3 tune d_head = 64 and d_head = 128 most heavily. Other values are padded up to the next compiled size or get less tuning. Scaling-law work also suggests model capacity gains from scaling come more from num_heads and d_model than from d_head. So there is no architectural pressure to grow d_head as models scale up. Standardizing on 128 keeps the kernel choice fixed while everything else scales.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the head count is set freely. In most models it is constrained by d_model = num_heads x d_head, so picking a head count fixes d_head and vice versa.
60 second bullets to scan on the way to the call.
The d_model = num_heads x d_head relationship
The 7B-class standard config (4096, 32, 128)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.