Llama-2 7B and Mistral 7B both use d_model = 4096 with 32 query heads, giving d_head = 128. Smaller 1-3B models typically use around 16 heads.
Picture splitting a wide highway into lanes. The highway has 4096 lanes total (that is d_model). Llama-2 7B and Mistral 7B both cut it into 32 lanes of lanes, so each one is 128 wide. That count, 32, is the number of attention heads. Smaller cars (1-3B models) make do with about 16 lanes of lanes because they have less traffic to route. The arithmetic is just d_model = num_heads x d_head, and 4096 = 32 x 128 is the standard split that lots of open-weight LLMs settled on.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The 7B-class LLM configuration converged on a specific set of numbers between 2023 and 2024: d_model = 4096, num_heads = 32, d_head = 128. Both Llama-2 7B and Mistral 7B use this exact configuration, and most subsequent 7B open-weight LLMs followed. The convergence is not coincidence; it reflects a small set of hard constraints (the d_model = num_heads x d_head identity), kernel efficiency considerations (FlashAttention prefers d_head = 64 or 128), and scaling-law guidance (d_model around 4096 for the 7B parameter target).
This deep dive walks the constraint that links the three numbers, why each value is what it is, how the pattern shifts at smaller and larger scales, why d_head = 128 has become so standardized, and how to read a model config.json correctly when GQA decouples query heads from KV heads.
The constraint: d_model = num_heads x d_head
Multi-head attention reshapes a flat d_model vector into a 2D (num_heads, d_head) block before the per-head matmul. The reshape requires d_model = num_heads * d_head, with strict integer divisibility.
Why the identity holds
The Q, K, V projections produce tensors of shape (batch, seq_len, d_model). To split into heads, the implementation views this as (batch, seq_len, num_heads, d_head). The view operation requires the trailing dimensions to multiply back to d_model. If they do not, PyTorch (or any framework) errors out at the reshape.
What this means for choosing numbers
You get two degrees of freedom, not three. Pick d_model and num_heads, and d_head follows. Pick d_model and d_head, and num_heads follows. Pick num_heads and d_head together, and you have fixed d_model.
The convention in modern LLMs is to pick d_model first (based on parameter-count targets and scaling laws), then pick d_head from a small kernel-friendly set ({64, 128}), and let num_heads emerge.
A worked example
For a 7B target with d_model = 4096 and d_head = 128:
num_heads = d_model / d_head = 4096 / 128 = 32
For a 70B target with d_model = 8192 and d_head = 128:
num_heads = 8192 / 128 = 64
The head dim stays constant; everything else scales.
The constraint is non-negotiable. Picking incompatible values produces a runtime error at model construction; you cannot work around it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama-2 7B config: hidden_size = 4096, num_attention_heads = 32, head_dim = 128.
- Mistral 7B config: hidden_size = 4096, num_attention_heads = 32, head_dim = 128 (identical to Llama-2 7B).
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy has d_head = 128 become so standard across model sizes?
Kernel efficiency dominates. FlashAttention v2 and v3 ship first-class tile configurations for d_head = 64 and d_head = 128. Other values fall back to less-tuned code paths. Scaling-law work also suggests model capacity gains from scaling come more from num_heads and d_model than from d_head, so there is no architectural pressure to grow d_head as models scale up. Standardizing on 128 keeps the kernel choice fixed while everything else scales.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the head count is set freely. It is constrained by d_model = num_heads x d_head, so picking a head count fixes d_head and vice versa, and you cannot pick arbitrary combinations.
60 second bullets to scan on the way to the call.
The d_model = num_heads x d_head relationship
The 7B-class standard config (4096, 32, 128)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.