Modern 7B-class LLMs, fill in typical head counts and the resulting per-head dimension

**At the 7B scale, the standard configuration is `d_model = 4096`, `num_heads = 32`, `d_head = 128`.** Both Llama-2 7B and Mistral 7B use exactly this. The arithmetic that ties the three numbers together is `d_model = num_heads * d_head`, which gives `4096 = 32 * 128`. Why this specific split: - **d_head = 128** is the head dimension that most attention kernels are tuned for. FlashAttention's tile sizes are picked for d_head = 64 or 128, and 128 is the common choice at 7B+ scale. - **num_heads = 32** drops out once d_model and d_head are picked. - **d_model = 4096** is the residual stream width for the 7B class. Smaller models in the 1-3B range typically drop num_heads to around 16. A 1.5B model might use `d_model = 2048`, `num_heads = 16`, `d_head = 128`, keeping the head dimension at 128 but halving both d_model and the head count. You cannot pick the three numbers independently. The constraint `d_model = num_heads * d_head` always has to hold, and `d_model % num_heads == 0` is enforced by the attention implementation.

**Standard 7B configuration**: `d_model = 4096`, `num_heads = 32`, `d_head = 128`. Both Llama-2 7B and Mistral 7B use this. The relationship is: $$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}$$ With `4096 / 32 = 128`, you get the canonical 7B-class head dimension. The choices are not all free. `d_model` is picked first based on model capacity targets; `d_head` is picked based on kernel efficiency (most FlashAttention configurations are tuned for d_head = 64 or 128); `num_heads` falls out as the quotient. **Smaller models** in the 1-3B range: - TinyLlama 1.1B: `d_model = 2048`, `num_heads = 32`, `d_head = 64`. - Llama 3.2 1B: `d_model = 2048`, `num_heads = 32`, `d_head = 64` (with GQA: 8 KV heads). - A typical 1.5B-3B config: `d_model = 2048-2560`, `num_heads = 16` typically, `d_head = 128`. The pattern at smaller scale varies more than at 7B. Some models keep d_head = 128 and reduce num_heads to 16; others keep num_heads = 32 and reduce d_head to 64. Both are common, and the choice depends on whether the team prioritizes kernel reuse (d_head = 128 wins) or per-head specialization (more heads with smaller d_head wins). **Larger models** push num_heads higher. Llama-2 70B uses `d_model = 8192, num_heads = 64, d_head = 128`. Llama 4 Maverick and Mistral Large 3 continue scaling num_heads while keeping d_head at 128.