Explain scaled dot-product attention.

Scaled dot-product attention is the core operation inside every transformer. It lets each token in a sequence look at every other token and decide how much each one matters for the current computation. The inputs are three matrices: Q (queries), K (keys), and V (values). Each row of Q is one token asking a question. Each row of K is one token offering itself as a key, the way a label sits on a folder. Each row of V is the actual content that token contributes if it gets selected. The operation has three steps. First, compute the similarity between every query and every key by multiplying Q with K transposed. This gives a square matrix of raw scores. Second, divide those scores by the square root of `d_k`, the dimensionality of each key vector, and apply softmax along the key axis. That turns scores into proper probability weights that sum to one. Third, multiply those weights by V to get a weighted average. The result is one output vector per query, each one a blend of value vectors chosen by relevance. The scaling factor is the easy to miss part: without it, dot products grow with dimensionality and saturate the softmax, making gradients vanish during training.

Scaled dot-product attention is the operation that lets each token in a sequence selectively attend to every other token. The formula is the heart of every transformer: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ Three steps. Compute pairwise similarities with `QK^T`, which gives an `n by n` score matrix where entry `(i, j)` is how much query i resonates with key j. Apply the scaling and softmax to convert raw scores into normalised attention weights along the key axis. Multiply the weights by V to produce a weighted average of value vectors, one output per query. The scaling matters for a specific numerical reason. If queries and keys have unit-variance components, the dot product of two `d_k`-dimensional vectors has variance proportional to `d_k`. As `d_k` grows, large dot products push softmax into a saturated regime where one weight is near one and the rest are near zero. Gradients through saturated softmax are vanishingly small, so training stalls. Dividing by the square root of `d_k` rescales variance back to a constant, keeping gradients healthy.