The first blank is the square root of d_k, the per-head key dimension. The second blank is V, the value matrix being weighted by the attention distribution.
Imagine you have a question and a row of labelled boxes. You want to mix the contents of the boxes into one good answer. First, you compare your question card to every label and give each box a score. Big scores can shout over small ones, so you quiet them down by dividing by a fair amount tied to how detailed each label is. The square root part is just the right size of brake so no single label drowns out the rest. Now you have soft weights that say how much each box matters. But weights alone are not the answer. You still need to scoop the contents of each box and mix them by those weights. The contents are called V. The two missing pieces are the brake under the slash and the contents at the end.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Scaled dot-product attention works because of two choices: sqrt(d_k), not d_k, in the denominator, and V at the end, because softmax alone is not an output.
Blank one: the square root of d_k
The scale factor in the denominator is sqrt(d_k), where d_k is the per-head dimension of the key projection. Modern models typically pick d_k = 64 or d_k = 128. The scale is per head, not per model, because attention is computed independently in each head.
The reason is a variance argument. Suppose Q and K are initialised so that their components are roughly mean zero with unit variance, and components are approximately independent. Then the dot product of two d_k-dimensional vectors is a sum of d_k independent products. Each product has variance one, so the sum has variance d_k and standard deviation sqrt(d_k).
Softmax of inputs whose standard deviation grows with dimensionality saturates rapidly. One weight gets pushed near one, the rest near zero, and the gradient through softmax goes to zero with them. Training stalls. Dividing by sqrt(d_k) cancels that growth exactly, so softmax sees inputs with constant variance regardless of head size.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The 'Attention Is All You Need' paper introduced exactly this formula with the same scaling factor that survives in every modern transformer.
- PyTorch's F.scaled_dot_product_attention hardcodes 1/sqrt(head_dim) as the scale, exposing it as an override only for niche use cases.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the constant sqrt(d_k) rather than d_k or log(d_k)?
Derive Var(q · k) = d_k for independent unit-variance components. Standard deviation, not variance, is the right scale to divide by, and the standard deviation of the sum is the square root of the variance. That fixes the constant uniquely.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Writing d_k instead of sqrt(d_k), or forgetting V altogether. The square root specifically cancels the standard deviation growth of dot products, and V is what the attention weights are blending.
60 second bullets to scan on the way to the call.
Write the full formula including the scale factor and the final value multiplication.
Define d_k as the per-head dimension of the key projection.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.