sqrt(d_k), the per-head key dimension. Not sqrt(d_model) and not sqrt(num_heads). It keeps pre-softmax score variance near 1.
Picture a dart competition where each round you sum up scores from many dartboards at once. The more boards you add, the wider the range of total scores swings, just from random luck. To keep the leaderboard meaningful round to round, you divide the total by something that scales with how many boards you summed over, so a typical total stays around the same size. Attention does the same thing: the dot product between query and key is a sum over d_k terms, and dividing by the square root of d_k keeps the typical magnitude steady regardless of how big you make a single head.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The scalar is sqrt(d_k), the per-head key dimension. It is one of the cleanest examples of a deep learning constant chosen by first-principles math rather than empirical tuning, and it has held up across nine years and three orders of magnitude of model scale.
This deep dive derives the choice from the variance of a d_k-term dot product, walks why softmax cares about input scale, contrasts the right divisor against the obvious wrong candidates, and shows how the scaling interacts with modern attention modifications (RoPE, ALiBi, quantization, FlashAttention).
Mental model: sqrt(d_k) is a variance invariant, not a hyperparameter. It keeps the pre-softmax score distribution at standard deviation ≈ 1 regardless of head size.
The variance derivation
The choice of sqrt(d_k) is not arbitrary, it falls out of a one-line probability calculation.
Setup
Assume q, k in R^{d_k} with components drawn i.i.d. zero-mean unit-variance. This is approximately true after standard initialization and layer norm, which is the regime the formula was designed for.
The dot-product variance
The dot product is q · k = sum_{i=1}^{d_k} q_i k_i. Each summand q_i k_i has expectation zero by symmetry. By independence:
Var(q_i k_i) = E[q_i^2] E[k_i^2] - E[q_i k_i]^2 = 1 · 1 - 0 = 1Var(q · k) = sum_i Var(q_i k_i) = d_kStd(q · k) = sqrt(d_k)
Dividing q · k by sqrt(d_k) brings the standard deviation back to 1.
The display form
Key insight: the divisor is not a tunable constant. It is the only choice that makes the score standard deviation invariant to d_k.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Candidate divisor | Effect on score variance | Softmax behavior |
|---|---|---|
| 1 (no scaling) | Variance grows as d_k | Saturates at large d_k, near one-hot, gradient vanishes off-peak |
| sqrt(d_k) | Variance ≈ 1 regardless of d_k | Stable across head sizes; the canonical choice |
| d_k | Variance ≈ 1/d_k | Over-corrects; softmax never sharpens, attention is near-uniform |
| sqrt(d_model) | Variance ≈ d_k / d_model (wrong scale) | Wrong magnitude per head; under-corrects for multi-head |
Real products, models, and research that use this idea.
- Vaswani et al. 2017, equation 1: explicitly motivates sqrt(d_k) via the variance argument in section 3.2.1.
- Llama 4 Maverick, Llama 3 70B: d_k = 128, divisor = sqrt(128) ≈ 11.3.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the scaling specifically sqrt(d_k) rather than d_k or 1/d_k?
The variance of a sum of d_k unit-variance products is d_k, so the standard deviation is sqrt(d_k). Dividing by sqrt(d_k) normalizes the standard deviation back to 1, the natural unit for softmax inputs. d_k over-corrects (variance becomes 1/d_k, softmax never sharpens); 1/d_k has no statistical justification.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Writing sqrt(d_model) instead of sqrt(d_k). The scaling is per-head, so d_k = hidden_size / num_heads is the right value.
60 second bullets to scan on the way to the call.
Why the divisor is sqrt(d_k) specifically, not d_k or 1/d_k
Walk through the variance derivation that motivates the sqrt(d_k) divisor
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.