What specifically goes wrong if you remove the √d_k scaling from attention?

The √d_k scaling exists for one specific training-stability reason: without it, the model can't learn. The failure mode is not numerical: it's gradient flow through softmax. Here's the chain of reasoning step by step. First, the dot product Q_i · K_j is a sum of d_k products. If Q and K components are roughly unit variance, the variance of the sum is d_k. So the typical magnitude of an entry in QKᵀ grows as √d_k. For d_head = 128 (typical for modern LLMs), a typical entry is around 11 in absolute value. Second, feeding large values into softmax produces a peaked output. softmax({0, 0, 11, 0}) is roughly {0.00002, 0.00002, 0.99994, 0.00002}. The largest entry takes almost all the probability mass; the rest collapse toward zero. Third, look at what happens in the backward pass. The softmax Jacobian at a peaked output is near-zero off the peak: nudging a small entry barely changes its tiny probability. So gradients can't flow back to non-peak positions. Fourth, the model is stuck. It can't learn to attend somewhere new because the gradient signal to do so is essentially zero. Training stalls. Dividing by √d_k brings the variance back to 1, keeping softmax inputs in a range where the output stays soft and gradients flow to all positions: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ That's why every modern attention implementation includes the scaling.

The trap here is that the failure is a gradient flow problem, not a numerical one, people reach for FP16 overflow and get the wrong mechanism. The variance of a QKᵀ entry scales linearly with d_k, so the standard deviation scales as √d_k. For typical head dimensions like 64 or 128, that's large enough that unscaled scores saturate softmax: one position grabs essentially all the probability mass. Once softmax is saturated, the softmax Jacobian goes near-zero off the peak. Gradients can't redistribute attention to other positions, because nudging a near-zero probability barely changes it. Training stalls almost immediately on any non-trivial head dim. The scaling normalizes the variance back to 1: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ The other options miss the mechanism. FP16 overflow is secondary: gradient failure happens long before NaNs. Softmax always normalizes by construction, so the output is still a probability distribution, just a degenerate one. And multi-head parallelism is completely orthogonal to scaling. The correct answer is the gradient flow story.