Why is softmax used in attention rather than alternatives like sparsemax or simple sum normalization?

Attempt it

Quick reference

TL;DR

Softmax is smooth and never outputs an exact zero, so gradients always reach every input position; sparsemax kills gradient at the zeros and blocks learning there.

Memory aid

Easy to grasp

Picture rating every restaurant in town on a sliding scale where every place gets at least a tiny score, never exactly zero. If you change your mind tomorrow about a place you didn't like, you can nudge its score up from that tiny number. Now picture a stricter system: only your top three get a score and everyone else is a hard zero. Once a restaurant is a hard zero, you have no way to change your mind about it later. The system has effectively forgotten how to reconsider it. A learning model needs the sliding system. It is always adjusting which words to pay attention to, and 'always a tiny score' means it can always reconsider any word.

Key concepts

Softmax Sparsemax Differentiability and gradient flow

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Lay out softmax's three properties, contrast with sparsemax's interpretability vs trainability tradeoff, explain why exponential amplification beats linear normalization, and mention α-entmax as the interpolating family.

Side-by-side

Normalizer	Always positive?	Exponential amp?	Gradient flow?	Used in production?
Softmax	Yes	Yes	All positions	Universal
Sparsemax	No (exact zeros)	Linear	Dead at zeros	Rare
Sum norm (x/Σx)	Requires non-neg input	Linear	All positions if positive	Almost never
α-entmax	Configurable	Configurable	Partial	Some research

Real-world examples

Real products, models, and research that use this idea.

Martins & Astudillo 2016 'From Softmax to Sparsemax' introduced sparsemax and demonstrated its interpretability benefits on NLP tasks.
Peters et al. 2019 'Sparse Sequence to Sequence Models' used α-entmax to interpolate between softmax and sparsemax in seq2seq attention.
Every production transformer (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Llama 4 Maverick, DeepSeek V4) uses standard softmax attention.
The 'attention is the gradient of LSE' identity is what enables FlashAttention's online softmax algorithm, exploiting the algebraic structure of softmax that other normalizers lack.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

Qα-entmax interpolates between softmax (α=1) and sparsemax (α=2). What's the intuition for α as a 'softness' hyperparameter?

QAre there scenarios where softmax's exponential amplification HURTS, i.e. where you'd want a less peaked normalizer?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying 'softmax sums to 1' as the reason. Many normalizers sum to 1; that property isn't what makes softmax the right pick for trainable attention.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.