Dividing pre-softmax attention scores by an extra factor > 1 at inference does what?

Softmax converts raw scores into a probability distribution. Its behavior is sensitive to the magnitude of the input scores: large differences between scores produce a peaked distribution (almost all mass on the top score), small differences produce a flat distribution (mass spread across many scores). Dividing all scores by a number greater than 1 before softmax shrinks the differences between them. The top score is still the largest, but the gap to the runner-ups is smaller. After softmax, the resulting attention weights are flatter, more mass goes to keys that previously got little weight. This is sometimes called attention temperature greater than 1 (by analogy to sampling temperature in language model decoding). It's used in long-context inference because some models, when stretched to context lengths well beyond what they were trained on, start showing attention patterns where almost all the weight collapses onto the few most recent tokens. Distant content is effectively ignored. Softening the attention distribution at inference time, without retraining, can recover some of that distant attention. The opposite move (multiplying scores by greater than 1, or dividing by less than 1) sharpens the distribution, making it more peaked. That's rarely useful at inference because trained attention is usually already peaked enough; sharpening tends to make the model brittle.

**The direction matters.** Dividing pre-softmax scores by a factor greater than 1 softens attention. Multiplying (or dividing by less than 1) sharpens it. The mechanism is simple. Softmax exponentiates differences in input scores. If the top score beats the next-best by 5 units, softmax assigns roughly e^5 = 148 times more weight to the top. If you divide all scores by 2, the gap becomes 2.5 units, the weight ratio drops to about 12x. The peak is still there but flatter. **Why this comes up in long-context inference.** Models trained at shorter context (say 4k tokens) sometimes show attention collapse when run at 32k+ tokens. Attention concentrates on the last few hundred tokens; distant content in the prompt gets near-zero weight. The standard explanation is that the dot-product magnitudes scale with sequence length while the softmax temperature stays fixed, so the effective distribution becomes too peaked. Dividing the scores by a length-dependent factor counteracts this. Production usage shows up in length-extension recipes. **YaRN** (Yet Another RoPE extensioN), used by Mistral and Llama variants, includes a temperature term that softens attention as a function of context length. The Mistral team's NTK-aware scaling does something similar. The values are typically small, 1.05 to 1.5, applied during long-context inference only. **The risk is over-softening.** Push the factor too high and attention becomes near-uniform, which destroys the model's ability to focus on the relevant tokens. The right value is tuned empirically per model and context length.