Why is softmax used in attention rather than alternatives like sparsemax or simple sum normalization?

**Softmax** is not the only way to turn a vector of scores into a probability distribution. So why is it the universal choice in attention? $$\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$ Three properties matter together: - **Always positive, no exact zeros.** Every position gets at least a tiny weight, and that tiny weight carries a non-zero gradient back to the input. The model can always re-learn to attend to any position. - **Smooth and differentiable everywhere.** Ideal for backprop. - **Exponential amplification.** Small score differences produce large weight differences. The model can produce sharp distributions when one position is clearly right and soft ones when things are ambiguous. **Sparsemax** produces sparse distributions with exact zeros. Nice for interpretability, but at any zero position, the gradient is also zero. Once rejected, that position stays rejected. **Sum normalization** (`x/Σx`) requires non-negative inputs and has only linear amplification, no way to produce sharp attention without enormous score magnitudes. The other MCQ options are red herrings: many normalizers sum to 1, complexity is O(n) for all of them. The distinguishing property is the combination 'always soft, always differentiable, exponentially flexible'.

Three properties together make **softmax** the right primitive for attention: $$\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$ Smooth and differentiable everywhere. Strictly positive output, no exact zeros, so gradients reach every position every step. Exponential amplification that lets attention go sharp when one position is clearly right and stay soft when several positions are plausible. Sparsemax sacrifices the first two for interpretable sparsity. Its exact zeros block gradient flow: once a position has weight zero, the partial derivative there is also zero, and the model has no signal to learn to attend there. Sum normalization (`x/Σx`) lacks exponential amplification and requires non-negative inputs, which dot products don't provide. The combination 'always soft, always differentiable, exponentially flexible' is what end to end gradient training of attention needs. The trap on this question is reaching for 'sums to 1' as the discriminator, many normalizers do, that property isn't what makes softmax special. Hence universal adoption in every production transformer (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Llama 4 Maverick, DeepSeek V4).