Complete the canonical attention formula: Attention(Q, K, V) = _(QKᵀ / _) · V

Fill in blank·Easy·4.0 · 0·~1 min·Asked atEy Salesforce Weaviate·Relevant atMicrosoft

Attempt it

Attention(Q, K, V) = (QKᵀ / ) · V

TL;DR

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V. The softmax turns raw scores into weights; the √d_k stabilizes the variance so softmax doesn't saturate.

Memory aid

Easy to grasp

Picture a chef tasting a row of sauces to decide how much of each to ladle into a final stew. Each sauce gets a raw 'matches my dish' score, but those scores can be wildly different in scale, so the chef shrinks them down to a fair size first. Then the chef turns the scores into percentages that add up to 100%, and finally pours each sauce into the stew in those proportions. That whole tasting and mixing routine is what the attention formula does.

Key concepts

Scaled dot product attention Softmax Variance scaling (√d_k)

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Equation 1 of Vaswani et al. 2017 is the most-recited formula in modern deep learning.

\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Knowing it cold, including the shape flow and the rationale for each piece, is the bare minimum for any interview that touches attention. But the surface formula hides three substantial mathematical questions: why the dot product specifically (and not additive scoring), why the square-root scaling (and not d_k or 1/d_k), and why softmax (and not sparsemax or argmax). This deep dive answers all three, then walks the modern implementation landscape, FlashAttention's blockwise computation, the additive modifications (masks, ALiBi, RoPE), and where approximation variants give up exactness for scalability.

The four-step recipe

Reading the formula left to right is reading the four-step recipe attention executes for every output position.

Score, scale, softmax, sum

Score, QKᵀ produces an n×n compatibility matrix. Each entry (i, j) is q_i · k_j.
Scale, divide by √d_k to keep score variance ≈ 1 regardless of how large d_k is.
Softmax: apply row-wise over the keys axis. Each row becomes a probability distribution; the row entry (i, j) is the attention weight from query i to key j.
Sum, multiply by V to take a weighted aggregation of value vectors. The output for each query position is a convex combination of all value vectors.

What the output represents

Output shape is (n, d_v): one context-aware vector per query position. Every output is a soft retrieval over the entire input sequence, with retrieval weights determined by content-based query key compatibility.

The recipe is identical whether attention is causal (with mask), bidirectional (no mask), cross-attention (Q from decoder, K/V from encoder), or self-attention (Q/K/V all from same input).

\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Why √d_k specifically

Shape flow and multi-head generalization

FlashAttention and the modern implementation landscape

Approximation variants and what they give up

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import torch, torch.nn.functional as F

def attention(Q, K, V):
    # Q, K: (..., n, d_k); V: (..., n, d_v)
    d_k = Q.size(-1)
    scores = (Q @ K.transpose(-2, -1)) / (d_k ** 0.5)
    weights = F.softmax(scores, dim=-1)  # row-wise over keys
    return weights @ V

Real products, models, and research that use this idea.

Equation 1 of Vaswani et al. 2017, the foundational definition of attention used in every transformer since.
FlashAttention computes this exact formula with O(n) HBM memory by tiling and fused kernels.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the scaling specifically √d_k rather than d_k or 1/d_k?

Variance of a sum of d_k unit variance products is d_k, so standard deviation is √d_k. Dividing by √d_k normalizes the standard deviation back to 1, which is the natural unit for softmax inputs. d_k would over-correct (variance 1/d_k → softmax never sharpens); 1/d_k makes no statistical sense.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Forgetting the √d_k or putting d_k instead of √d_k: the square root matters. Or applying softmax over the wrong axis.

60 second bullets to scan on the way to the call.

Write the formula from memory exactly
Shape of QKᵀ for sequences of length n

Primary sources. Browse if you want the original framing.