Pick what happens to attention weights when pre-softmax scores are divided by a temperature T > 1

**The correct answer is B: the weights become more uniform.** Softmax with temperature T computes: $$w_j = \frac{e^{s_j / T}}{\sum_k e^{s_k / T}}$$ When `T > 1`, every score is divided by a number greater than 1, shrinking it toward 0. After exponentiation, the ratios between scores narrow, and the resulting distribution flattens. Mass spreads across more positions. **The two limits.** - `T -> infinity`: every `s_j / T -> 0`, every `exp(0) = 1`, the row becomes uniform `(1/T_k, ..., 1/T_k)`. - `T -> 0`: gaps widen unboundedly, one position dominates, the row becomes one-hot at the argmax. For `T > 1` but finite, the distribution is somewhere between the original and uniform: softer but not flat. **The default in attention.** The sqrt(d_k) divisor in scaled dot-product attention is a built-in temperature of T = sqrt(d_k). It is chosen specifically to keep score variance near 1 so softmax neither saturates (T too small) nor collapses to uniform (T too large). **Why the other options are wrong.** - **Option A:** describes T < 1 behavior, not T > 1. - **Option C:** uniform happens only in the limit T -> infinity, not at any finite T > 1. - **Option D:** softmax is shift-invariant (add a constant to all inputs and outputs are unchanged), but NOT scale-invariant. Dividing by T > 1 changes the distribution shape.

**The weights become more uniform (option B).** Softmax with temperature T applies the formula: $$w_j = \frac{e^{s_j / T}}{\sum_k e^{s_k / T}}$$ When `T > 1`, every score is shrunk toward zero. The exponential then narrows the ratios between scores, and the post-softmax distribution flattens. **The intuition via limits.** - **As `T -> infinity`:** every `s_j / T -> 0`, every `exp(s_j / T) -> 1`, the row converges to uniform `(1/T_k, ..., 1/T_k)`. - **As `T -> 0+`:** gaps between scores explode, the largest score dominates, the row converges to one-hot at the argmax. - **At `T = 1`:** standard softmax with no temperature adjustment. For `T > 1` but finite, the distribution sits between the original (at T = 1) and uniform (at T = infinity): the same ranking is preserved but the relative magnitudes are compressed. **Why softmax is not scale-invariant.** Softmax is SHIFT-invariant: subtracting a constant from every input leaves the output unchanged (the constant cancels from numerator and denominator). But it is NOT scale-invariant: dividing every input by T > 1 reshapes the distribution because the exponential function is not linear. **Where temperature appears in attention.** The sqrt(d_k) divisor in scaled dot-product attention is exactly a built-in temperature of `T = sqrt(d_k)`. It is calibrated to keep pre-softmax score variance near 1 regardless of head size, so softmax stays in its responsive regime. Per-head temperature is rarely tuned beyond this default in standard transformers; it is more often manipulated at the LLM's output sampling layer, where T > 1 produces more diverse text and T < 1 produces more focused text.