The first blank is the square root of d_k, the per-head key dimension. The second blank is V, the value matrix being weighted by the attention distribution.
Imagine you have a question and a row of labelled boxes. You want to mix the contents of the boxes into one good answer. First, you compare your question card to every label and give each box a score. Big scores can shout over small ones, so you quiet them down by dividing by a fair amount tied to how detailed each label is. The square root part is just the right size of brake so no single label drowns out the rest. Now you have soft weights that say how much each box matters. But weights alone are not the answer. You still need to actually scoop the contents of each box and mix them by those weights. The contents are called V. The two missing pieces are the brake under the slash and the contents at the end.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The fill in the blank format is short, but the two blanks together cover the whole story of why scaled dot-product attention works at all. The first blank is where most candidates lose half marks by writing d_k instead of sqrt(d_k). The second blank is where some candidates lose the rest by forgetting that softmax alone is not an output.
This section walks through each blank with enough detail that the choice feels inevitable rather than memorised.
Blank one: the square root of d_k
The scale factor in the denominator is sqrt(d_k), where d_k is the per-head dimension of the key projection. Modern models pick d_k = 64 for Llama-class architectures and d_k = 128 for some larger variants. The scale is per-head, not per-model, because attention is computed independently in each head.
The reason is a variance argument. If Q and K are initialised so that their components are roughly mean zero with unit variance, and components are approximately independent, then the dot product of two d_k-dimensional vectors is a sum of d_k independent products. Each product has variance one, so the sum has variance d_k and standard deviation sqrt(d_k).
Softmax of inputs whose standard deviation grows with dimensionality saturates rapidly. One weight gets pushed near one, the rest near zero, and the gradient through softmax goes to zero with them. Training stalls. Dividing by sqrt(d_k) cancels that growth exactly, so softmax sees inputs with constant variance regardless of head size.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The 'Attention Is All You Need' paper introduced exactly this formula with the same scaling factor that survives in every modern transformer.
- PyTorch's `F.scaled_dot_product_attention` hardcodes `1/sqrt(head_dim)` as the scale, exposing it as an override only for niche use cases.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the constant `sqrt(d_k)` rather than `d_k` or `log(d_k)`?
Derive Var(q · k) = d_k for independent unit-variance components. Standard deviation, not variance, is the right scale to divide by, and the standard deviation of the sum is the square root of the variance. That fixes the constant uniquely.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Writing d_k instead of sqrt(d_k), or forgetting V altogether. The square root specifically cancels the standard deviation growth of dot products, and V is what the attention weights are blending.
60 second bullets to scan on the way to the call.
Write the full formula including the scale factor and the final value multiplication.
Define d_k as the per-head dimension of the key projection.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.