**The correct answer is B: the weights become more uniform.**
Softmax with temperature T computes:
$$w_j = \frac{e^{s_j / T}}{\sum_k e^{s_k / T}}$$
When `T > 1`, every score is divided by a number greater than 1, shrinking it toward 0. After exponentiation, the ratios between scores narrow, and the resulting distribution flattens. Mass spreads across more positions.
**The two limits.**
- `T -> infinity`: every `s_j / T -> 0`, every `exp(0) = 1`, the row becomes uniform `(1/T_k, ..., 1/T_k)`.
- `T -> 0`: gaps widen unboundedly, one position dominates, the row becomes one-hot at the argmax.
For `T > 1` but finite, the distribution is somewhere between the original and uniform: softer but not flat.
**The default in attention.** The sqrt(d_k) divisor in scaled dot-product attention is a built-in temperature of T = sqrt(d_k). It is chosen specifically to keep score variance near 1 so softmax neither saturates (T too small) nor collapses to uniform (T too large).
**Why the other options are wrong.**
- **Option A:** describes T < 1 behavior, not T > 1.
- **Option C:** uniform happens only in the limit T -> infinity, not at any finite T > 1.
- **Option D:** softmax is shift-invariant (add a constant to all inputs and outputs are unchanged), but NOT scale-invariant. Dividing by T > 1 changes the distribution shape.
**The weights become more uniform (option B).**
Softmax with temperature T applies the formula:
$$w_j = \frac{e^{s_j / T}}{\sum_k e^{s_k / T}}$$
When `T > 1`, every score is shrunk toward zero. The exponential then narrows the ratios between scores, and the post-softmax distribution flattens.
**The intuition via limits.**
- **As `T -> infinity`:** every `s_j / T -> 0`, every `exp(s_j / T) -> 1`, the row converges to uniform `(1/T_k, ..., 1/T_k)`.
- **As `T -> 0+`:** gaps between scores explode, the largest score dominates, the row converges to one-hot at the argmax.
- **At `T = 1`:** standard softmax with no temperature adjustment.
For `T > 1` but finite, the distribution sits between the original (at T = 1) and uniform (at T = infinity): the same ranking is preserved but the relative magnitudes are compressed.
**Why softmax is not scale-invariant.** Softmax is SHIFT-invariant: subtracting a constant from every input leaves the output unchanged (the constant cancels from numerator and denominator). But it is NOT scale-invariant: dividing every input by T > 1 reshapes the distribution because the exponential function is not linear.
**Where temperature appears in attention.** The sqrt(d_k) divisor in scaled dot-product attention is exactly a built-in temperature of `T = sqrt(d_k)`. It is calibrated to keep pre-softmax score variance near 1 regardless of head size, so softmax stays in its responsive regime. Per-head temperature is rarely tuned beyond this default in standard transformers; it is more often manipulated at the LLM's output sampling layer, where T > 1 produces more diverse text and T < 1 produces more focused text.
Quick reference
TL;DR
T > 1 flattens the distribution: gaps between scores shrink, attention spreads across more positions. In the limit T -> infinity the row becomes uniform.
Imagine a vote where everyone shouts a number out of 100 for each candidate. If you shrink everyone's volume to a whisper, the loudest shouter no longer drowns out the others and the final vote spreads out instead of going to a single landslide winner. Dividing the model's raw scores by a temperature greater than 1 is exactly that volume knob: it shrinks every number toward zero, so the final share of attention across all the items in the row evens out instead of being captured by one peak. Turn the knob the other way and the loudest voice wins everything.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Per-layer attention temperature tuning at inference time Standard transformers bake temperature into the sqrt(d_k) divisor at architecture-time; per-inference attention temperature tuning is rarely beneficial and not part of standard serving stacks.
Hard attention scenarios where you want exact zeros Temperature smoothly interpolates between argmax and uniform but cannot produce true sparsity. Use sparsemax or top-k thresholding if you need exact zeros across some keys.
Comparing model accuracy by tuning attention temperature post-hoc Attention temperature is part of the architecture the model was trained with; changing it at inference time shifts the model out of its training distribution and usually degrades quality.
Studying scale invariance properties Softmax is shift-invariant, not scale-invariant; temperature manipulation explicitly exploits the lack of scale invariance, so reasoning about it should not appeal to a non-existent invariance.
Quick scan · 2–4 min
Everything important, quickly.
State that T > 1 flattens the distribution, derive the two limits (T -> infinity uniform, T -> 0 argmax), explain shift-invariance vs scale-non-invariance, and connect to the sqrt(d_k) built-in temperature and LLM output sampling temperature.
Real-world examples
Real products, models, and research that use this idea.
1Vaswani et al. 2017: section 3.2.1 explains the sqrt(d_k) divisor as a built-in temperature chosen to keep pre-softmax variance near 1.
2LLM sampling at inference time: T = 0.7 (slightly sharper than default) is a common chat-model setting; T > 1 produces more diverse text; T -> 0 approaches greedy decoding.
3Hinton et al. 2015 (Distilling the Knowledge in a Neural Network): teacher logits divided by T > 1 to produce softer distributions for student training; the dark-knowledge insight.
4OpenAI, Anthropic, and Google APIs all expose a temperature parameter for chat completions, applied to the final vocabulary softmax during decoding.
5Modern decoding strategies like nucleus sampling and top-k filtering operate alongside temperature, often with T in [0.5, 1.5] depending on use case.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is softmax shift-invariant but not scale-invariant, and what is the practical implication?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the entropy of the attention weight distribution look like as a function of T?
Sign in or go Pro to see the approach for this follow-up.
QWhy is per-layer attention temperature rarely tuned at inference time, but LLM output sampling temperature is tuned per-request?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming softmax is scale-invariant. It is shift-invariant (subtract a constant from every input and weights are unchanged), not scale-invariant; dividing by T > 1 reshapes the distribution.