Explain scaled dot product attention, and why the scaling factor is used.
Scaled dot-product attention computes softmax of QK transposed divided by the square root of d_k, then weights V. The scaling keeps the softmax in a stable gradient regime.
Imagine a room full of labelled boxes, each holding some content. You walk in with a question written on a card. You compare your card to every label and decide how relevant each box is. Then you take a mix of their contents, weighted by how relevant they are. The query is your card, the keys are the labels, the values are the contents, and the mix is the attention output. The square root divisor is a temperature knob. It keeps the comparison fair when the cards are very detailed. Without it, one label can shout so loudly that all the others go silent, and you would never get a useful mix.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Scaled dot-product attention is one of the core pieces of modern machine learning. Each query scores every key with a dot product, the scores are scaled and passed through softmax, and the weights average the values.
Q, K, and V are three views of the same input
In self-attention, the inputs Q, K, and V are all derived from the same source sequence X by three separate linear projections. If X has shape [n, d_model], then Q = X W_Q, K = X W_K, and V = X W_V each have shape [n, d_k] or [n, d_v] depending on which projection. The three matrices W_Q, W_K, W_V are learned parameters.
The metaphor is a content-addressed lookup. Q is what each token is asking. K is how each token advertises itself, in the same vector space as Q so they can be compared by dot product. V is what each token contributes if its key wins. The fact that all three come from the same X is what makes self-attention content-addressed. A token can decide which other tokens to look at based purely on what is in the sequence, with no external index.
In cross-attention the only change is that K and V come from a different source than Q. Encoder-decoder models like the original Transformer use this for the decoder to attend to the encoder output.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- PyTorch's torch.nn.functional.scaled_dot_product_attention (added in 2.0) ships the formula as a fused kernel that dispatches to FlashAttention 2 (since 2.2) when available.
- Hugging Face Transformers calls the same operation inside every attention layer of Llama, Qwen, and DeepSeek implementations.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the softmax need normalisation along the key axis specifically?
Trace one row of the score matrix. Each row corresponds to one query and contains its similarity to every key. Softmax along that row turns the row into a distribution over keys, which is what the weighted sum of V requires.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reciting the formula without justifying the scale factor. The square root of d_k is not cosmetic. It is the variance correction that keeps gradients from vanishing as dimensionality grows.
60 second bullets to scan on the way to the call.
State the formula in full, including the scale factor and the softmax position.
Explain the role of Q, K, and V as three projections of the input.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.