sqrt(d_k), where d_k is the per-head key dimension, same value as head_dim (= hidden_size / num_heads in standard configurations). Not sqrt(d_model) and not sqrt(num_heads). The division happens before softmax: scores = (QK^T) / sqrt(d_k). The reason is variance control: for d_k-dimensional unit-variance vectors, dot products have variance d_k, so without scaling the softmax inputs would saturate as d_k grows, collapsing the attention distribution to nearly one-hot and killing gradients on the other positions.
The scalar is **sqrt(d_k)**, the square root of the per-head key dimension.
In a standard transformer, `d_k = hidden_size / num_heads`. For Llama 3 70B that is `8192 / 64 = 128`, so the divisor is `sqrt(128) ≈ 11.3`. The division is applied to the raw dot product before softmax:
$$\text{scores} = \frac{QK^\top}{\sqrt{d_k}}$$
The reason for this specific scalar is statistical. If Q and K components have roughly unit variance, the dot product `q · k = sum over d_k terms` has variance `d_k` and standard deviation `sqrt(d_k)`. Without scaling, raw scores would grow with d_k, and softmax would saturate at large d_k: one position dominates, the rest get near-zero weight, and gradients vanish almost everywhere.
Dividing by `sqrt(d_k)` brings the score variance back to roughly 1, regardless of head size. That is why models with d_k = 64, 128, or 192 all train with the same optimizer settings: the scaling is a variance invariant, not a tunable hyperparameter.
**`sqrt(d_k)`**, the square root of the per-head key dimension. The formula is:
$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
Three things to know cold for an interview.
**It is per-head, not per-model.** For Llama 3 70B, `d_k = hidden_size / num_heads = 8192 / 64 = 128`, so the divisor is `sqrt(128) ≈ 11.3`, not `sqrt(8192)`. The 'scaled' in scaled dot-product attention refers to this one specific scalar.
**The justification is a variance argument.** If Q and K components are zero-mean unit-variance, then `q · k = sum of d_k unit-variance products` has variance `d_k` and standard deviation `sqrt(d_k)`. Dividing by `sqrt(d_k)` restores unit variance. Without it, raw scores grow with d_k, softmax saturates, and the attention distribution collapses to nearly one-hot, killing gradients on the non-peak positions.
**It is a variance invariant, not a tuned hyperparameter.** That is what lets d_k = 64 (BERT), 128 (Llama, Mistral), and 192 (some MoE variants) all train with the same optimizer settings. Vaswani et al. specifically called out in 2017 that without this scaling, large-d_k models fail to train past the first few epochs.
Quick reference
TL;DR
sqrt(d_k), the per-head key dimension. Not sqrt(d_model) and not sqrt(num_heads). It keeps pre-softmax score variance near 1.
Picture a dart competition where each round you sum up scores from many dartboards at once. The more boards you add, the wider the range of total scores swings, just from random luck. To keep the leaderboard meaningful round to round, you divide the total by something that scales with how many boards you summed over, so a typical total stays around the same size. Attention does the same thing: the dot product between query and key is a sum over d_k terms, and dividing by the square root of d_k keeps the typical magnitude steady regardless of how big you make a single head.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Quantized attention with int4 or int8 K projections that break the unit-variance assumption The variance argument behind sqrt(d_k) assumes K components are roughly unit-variance; aggressive quantization can shift K variance, so an additional rescale may be needed to keep softmax in its non-saturated regime.
Asymmetric attention scoring where Q and K use different normalizations If one of Q or K is layer-normed and the other is not, their component variances differ from 1, and the simple sqrt(d_k) divisor under or over-corrects. The variance derivation has to be redone.
Additive attention scoring (Bahdanau-style) Additive attention uses a learned feedforward to compute scores rather than a dot product, so the variance derivation does not apply and the sqrt(d_k) divisor is not used.
Linear or kernel attention variants like Performer Linear attention factors softmax through a feature map, and the per-head score variance has a different functional form; the divisor either changes or is absorbed into the feature map.
Quick scan · 2–4 min
Everything important, quickly.
State the divisor sqrt(d_k), derive it from the variance of a d_k-term dot product, contrast with wrong candidates (1, d_k, sqrt(d_model)), and explain why the scaling is invariant across head sizes.
Side-by-side
Candidate divisor
Effect on score variance
Softmax behavior
1 (no scaling)
Variance grows as d_k
Saturates at large d_k, near one-hot, gradient vanishes off-peak
sqrt(d_k)
Variance ≈ 1 regardless of d_k
Stable across head sizes; the canonical choice
d_k
Variance ≈ 1/d_k
Over-corrects; softmax never sharpens, attention is near-uniform
sqrt(d_model)
Variance ≈ d_k / d_model (wrong scale)
Wrong magnitude per head; under-corrects for multi-head
Real-world examples
Real products, models, and research that use this idea.
1Vaswani et al. 2017, equation 1: explicitly motivates sqrt(d_k) via the variance argument in section 3.2.1.
3Mistral 7B and Mixtral: d_k = 128 per attention head, same sqrt(128) scaling.
4Gemma 4 27B: head_dim = 256, so the divisor is sqrt(256) = 16.
5PyTorch's torch.nn.functional.scaled_dot_product_attention applies the sqrt(d_k) divisor automatically and dispatches to FlashAttention kernels under the hood.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the scaling specifically sqrt(d_k) rather than d_k or 1/d_k?
Sign in or go Pro to see the approach for this follow-up.
QHow does FlashAttention apply the sqrt(d_k) scaling inside its blockwise softmax computation?
Sign in or go Pro to see the approach for this follow-up.
QWhy does RoPE preserve the sqrt(d_k) scaling, but int8 KV-cache quantization sometimes break it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Writing sqrt(d_model) instead of sqrt(d_k). The scaling is per-head, so d_k = hidden_size / num_heads is the right value.