The full canonical formula for **scaled dot product attention** comes from equation 1 of Vaswani et al. 2017 ('Attention Is All You Need'):
$$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
Walking through it term by term:
- QKᵀ is the matrix of raw compatibility scores. Q has shape `(n, d_k)` and K has shape (n, d_k), so QKᵀ is (n, n). Each entry (i, j) is the dot product Q_i · K_j: how well query i matches key j.
- Dividing by √d_k is the 'scaled' part of scaled dot product attention. It normalizes the score variance so softmax doesn't saturate at large d_k.
- softmax is applied row-wise over the keys axis. Each row becomes a probability distribution: the attention weights.
- Multiplying by V (shape (n, d_v)) gives the output (n, d_v), where each row is a weighted sum of value vectors.
Final output shape: (n, d_v). For each input position, you've produced a context-aware output that's a weighted mixture of every position's value, with weights driven by query key compatibility. This formula is the literal definition of attention and shows up in every transformer architecture.
Equation 1 of Vaswani 2017, the most-recited formula in modern deep learning:
$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
Four steps, in order: **score, scale, softmax, sum**.
Score is `QKᵀ`, that produces an n×n matrix of raw query key dot products. Scale is `/ √d_k`, that normalizes the score variance so softmax doesn't saturate when `d_k` is large, which is the whole reason 'scaled' is in the name. Softmax is applied row-wise to turn each row of scores into a probability distribution over the keys. Sum is the multiplication by V, a weighted aggregation of value vectors using those probabilities as weights.
The shape flow: Q and K are `(n, d_k)`, V is `(n, d_v)`. `QKᵀ` is `(n, n)`. After softmax, still `(n, n)` but row-stochastic. Multiplying by V gives `(n, d_v)`.
The trade off worth mentioning: that `n×n` score matrix is the entire reason FlashAttention exists, Dao et al. 2022 showed you can compute the exact same output without materializing the full matrix in HBM, which is the modern standard for any serious attention implementation.
Quick reference
TL;DR
Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V. The softmax turns raw scores into weights; the √d_k stabilizes the variance so softmax doesn't saturate.
Picture a chef tasting a row of sauces to decide how much of each to ladle into a final stew. Each sauce gets a raw 'matches my dish' score, but those scores can be wildly different in scale, so the chef shrinks them down to a fair size first. Then the chef turns the scores into percentages that add up to 100%, and finally pours each sauce into the stew in those proportions. That whole tasting and mixing routine is what the attention formula does.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Sequences in the hundreds of thousands of tokens at training time The exact O(n²) compute in QKᵀ becomes the bottleneck even with FlashAttention's O(n) HBM trick. Linear-attention variants or sparse patterns (BigBird, Longformer) trade exactness for tractable scaling in this regime.
Very small d_k (< 16) with tied Q and K projections The variance argument behind √d_k loses its statistical justification, score distributions are no longer well-approximated by N(0, d_k), and softmax saturation behavior becomes regime-dependent. Use additive scoring (Bahdanau-style) or larger d_k instead.
Hard-realtime decoding on accelerators without efficient softmax kernels Standard softmax requires a max-reduction and an exp pass that don't always fuse well outside CUDA / ROCm. On some edge accelerators, hash-attention or sparsemax variants ship faster despite being mathematically inexact.
Strict deterministic / bit-reproducible inference across hardware Different attention kernels (FlashAttention v2, FlashAttention v3, memory-efficient, math) round softmax intermediates differently. The output is identical up to floating point tolerance but not bit for bit reproducible across kernels: a problem for some compliance regimes.
Quick scan · 2–4 min
Everything important, quickly.
Walk shape flow, justify the √d_k from variance, note the row-wise softmax, and mention FlashAttention as the modern exact-implementation.
Code example
python
import torch, torch.nn.functional as F
def attention(Q, K, V):
# Q, K: (..., n, d_k); V: (..., n, d_v)
d_k = Q.size(-1)
scores = (Q @ K.transpose(-2, -1)) / (d_k ** 0.5)
weights = F.softmax(scores, dim=-1) # row-wise over keys
return weights @ V
Real-world examples
Real products, models, and research that use this idea.
1Equation 1 of Vaswani et al. 2017, the foundational definition of attention used in every transformer since.
2FlashAttention computes this exact formula with O(n) HBM memory by tiling and fused kernels.
3PyTorch's torch.nn.functional.scaled_dot_product_attention implements exactly this formula and dispatches to FlashAttention or memory-efficient kernels under the hood.
4Every transformer in production, GPT-5.5, Llama 4, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4, runs this formula inside every attention block.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the scaling specifically √d_k rather than d_k or 1/d_k?
Sign in or go Pro to see the approach for this follow-up.
QFlashAttention computes the same formula but in O(n) memory. What's the trick?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting the √d_k or putting d_k instead of √d_k: the square root matters. Or applying softmax over the wrong axis.