Llama-2 7B and Mistral 7B both use d_model = 4096 with query heads, giving a head dimension d_head of . Smaller models in the 1-3B range typically use around query heads.
**At the 7B scale, the standard configuration is `d_model = 4096`, `num_heads = 32`, `d_head = 128`.** Both Llama-2 7B and Mistral 7B use exactly this. The arithmetic that ties the three numbers together is `d_model = num_heads * d_head`, which gives `4096 = 32 * 128`.
Why this specific split:
- **d_head = 128** is the head dimension that most attention kernels are tuned for. FlashAttention's tile sizes are picked for d_head = 64 or 128, and 128 is the common choice at 7B+ scale.
- **num_heads = 32** drops out once d_model and d_head are picked.
- **d_model = 4096** is the residual stream width for the 7B class.
Smaller models in the 1-3B range typically drop num_heads to around 16. A 1.5B model might use `d_model = 2048`, `num_heads = 16`, `d_head = 128`, keeping the head dimension at 128 but halving both d_model and the head count.
You cannot pick the three numbers independently. The constraint `d_model = num_heads * d_head` always has to hold, and `d_model % num_heads == 0` is enforced by the attention implementation.
**Standard 7B configuration**: `d_model = 4096`, `num_heads = 32`, `d_head = 128`. Both Llama-2 7B and Mistral 7B use this. The relationship is:
$$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}$$
With `4096 / 32 = 128`, you get the canonical 7B-class head dimension.
The choices are not all free. `d_model` is picked first based on model capacity targets; `d_head` is picked based on kernel efficiency (most FlashAttention configurations are tuned for d_head = 64 or 128); `num_heads` falls out as the quotient.
**Smaller models** in the 1-3B range:
- TinyLlama 1.1B: `d_model = 2048`, `num_heads = 32`, `d_head = 64`.
- Llama 3.2 1B: `d_model = 2048`, `num_heads = 32`, `d_head = 64` (with GQA: 8 KV heads).
- A typical 1.5B-3B config: `d_model = 2048-2560`, `num_heads = 16` typically, `d_head = 128`.
The pattern at smaller scale varies more than at 7B. Some models keep d_head = 128 and reduce num_heads to 16; others keep num_heads = 32 and reduce d_head to 64. Both are common, and the choice depends on whether the team prioritizes kernel reuse (d_head = 128 wins) or per-head specialization (more heads with smaller d_head wins).
**Larger models** push num_heads higher. Llama-2 70B uses `d_model = 8192, num_heads = 64, d_head = 128`. Llama 4 Maverick and Mistral Large 3 continue scaling num_heads while keeping d_head at 128.
Quick reference
TL;DR
Llama-2 7B and Mistral 7B both use d_model = 4096 with 32 query heads, giving d_head = 128. Smaller 1-3B models typically use around 16 heads.
Picture splitting a wide highway into lanes. The highway has 4096 lanes total (that is d_model). Llama-2 7B and Mistral 7B both cut it into 32 lanes of lanes, so each one is 128 wide. That count, 32, is the number of attention heads. Smaller cars (1-3B models) make do with about 16 lanes of lanes because they have less traffic to route. The arithmetic is just d_model = num_heads x d_head, and 4096 = 32 x 128 is the standard split that lots of open-weight LLMs settled on.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Models with custom head dimensions outside the kernel-friendly set Research architectures using d_head = 80, 96, or 256 break the typical configuration assumptions and require custom kernels or slow fallbacks.
Reading legacy GPT-2 or BERT configurations Earlier transformer families used different scaling patterns (GPT-2 small at d_model = 768, num_heads = 12, d_head = 64); applying the 7B pattern as a universal rule misleads.
MoE or MLA architectures where d_head no longer follows the standard relationship DeepSeek-style MLA decouples attention dimensionality from the standard d_model = num_heads x d_head identity; the typical numbers do not transfer.
Vision transformers (ViT) or other non-text transformer variants Vision and multimodal stacks pick d_head and num_heads based on different empirical curves; the LLM-class numbers do not necessarily apply.
Quick scan · 2–4 min
Everything important, quickly.
60s: state the 7B standard (d_model = 4096, num_heads = 32, d_head = 128), give the arithmetic identity, note both Llama-2 7B and Mistral 7B use the same config, mention smaller models drop to ~16 heads when keeping d_head = 128, explain why d_head = 128 is the kernel-preferred default.
Real-world examples
Real products, models, and research that use this idea.
3Llama-2 70B config: hidden_size = 8192, num_attention_heads = 64, head_dim = 128 (head dim held constant).
4TinyLlama 1.1B: hidden_size = 2048, num_attention_heads = 32, head_dim = 64 (head dim halved at small scale).
5Llama 4 Maverick and Mistral Large 3 continue the d_head = 128 convention across the size family.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy has d_head = 128 become so standard across model sizes?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens if you pick d_head = 80 or some other non-standard value?
Sign in or go Pro to see the approach for this follow-up.
QHow does GQA affect the way you read num_heads from a config?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the head count is set freely. It is constrained by d_model = num_heads x d_head, so picking a head count fixes d_head and vice versa, and you cannot pick arbitrary combinations.