Softmax converts raw scores into a probability distribution. Its behavior is sensitive to the magnitude of the input scores: large differences between scores produce a peaked distribution (almost all mass on the top score), small differences produce a flat distribution (mass spread across many scores).
Dividing all scores by a number greater than 1 before softmax shrinks the differences between them. The top score is still the largest, but the gap to the runner-ups is smaller. After softmax, the resulting attention weights are flatter, more mass goes to keys that previously got little weight.
This is sometimes called attention temperature greater than 1 (by analogy to sampling temperature in language model decoding). It's used in long-context inference because some models, when stretched to context lengths well beyond what they were trained on, start showing attention patterns where almost all the weight collapses onto the few most recent tokens. Distant content is effectively ignored. Softening the attention distribution at inference time, without retraining, can recover some of that distant attention.
The opposite move (multiplying scores by greater than 1, or dividing by less than 1) sharpens the distribution, making it more peaked. That's rarely useful at inference because trained attention is usually already peaked enough; sharpening tends to make the model brittle.
**The direction matters.** Dividing pre-softmax scores by a factor greater than 1 softens attention. Multiplying (or dividing by less than 1) sharpens it.
The mechanism is simple. Softmax exponentiates differences in input scores. If the top score beats the next-best by 5 units, softmax assigns roughly e^5 = 148 times more weight to the top. If you divide all scores by 2, the gap becomes 2.5 units, the weight ratio drops to about 12x. The peak is still there but flatter.
**Why this comes up in long-context inference.** Models trained at shorter context (say 4k tokens) sometimes show attention collapse when run at 32k+ tokens. Attention concentrates on the last few hundred tokens; distant content in the prompt gets near-zero weight. The standard explanation is that the dot-product magnitudes scale with sequence length while the softmax temperature stays fixed, so the effective distribution becomes too peaked. Dividing the scores by a length-dependent factor counteracts this.
Production usage shows up in length-extension recipes. **YaRN** (Yet Another RoPE extensioN), used by Mistral and Llama variants, includes a temperature term that softens attention as a function of context length. The Mistral team's NTK-aware scaling does something similar. The values are typically small, 1.05 to 1.5, applied during long-context inference only.
**The risk is over-softening.** Push the factor too high and attention becomes near-uniform, which destroys the model's ability to focus on the relevant tokens. The right value is tuned empirically per model and context length.
Quick reference
TL;DR
Dividing pre-softmax logits by a factor greater than 1 flattens the attention distribution, useful for redistributing weight away from over-peaked recent tokens at long context.
Picture a roomful of people voting on what to do next. If everyone's votes are very lopsided, the loudest few decide everything. Dividing every vote count by a number bigger than 1 before tallying brings everyone's totals closer together, so quieter voters get more say. The model works the same way when it picks which earlier words to focus on. Shrinking all the raw scores closer together before the final tally spreads its focus across distant words instead of pinning everything to the few loudest recent ones. That is exactly what you want when a long-prompt model has started ignoring useful information far back in the conversation.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Inference at or below the model's training context length The attention distribution is already calibrated; adding temperature scaling will flatten attention unnecessarily and degrade focus on relevant tokens.
Tasks that require razor-sharp attention on a single token (needle in haystack retrieval, exact code completion) Softening attention reduces the model's ability to commit to one token; for retrieval-style tasks the peaked distribution is the desired behavior.
Models that already include length-aware scaling in their architecture (e.g., trained natively at long context with YaRN scaling baked in) Adding more temperature scaling on top double-counts the correction; the model expects the scaled regime as its default.
When the quality drop at long context is caused by position encoding extrapolation failure rather than attention collapse Temperature scaling fixes a softmax-magnitude problem, not a position-encoding problem; if RoPE is out of its trained range, you need position interpolation or YaRN, not just temperature.
Quick scan · 2–4 min
Everything important, quickly.
5m: softmax sensitivity to logit magnitude, why long context causes attention collapse, YaRN-style temperature scaling formula and tuning, interaction with RoPE position interpolation, and when temperature scaling is the wrong fix.
Real-world examples
Real products, models, and research that use this idea.
1YaRN (Yet Another RoPE extensioN) by Peng et al. 2023 uses a temperature term in its length-extension recipe and is integrated into Mistral and Llama variants.
2Together AI's long-context Llama 3 70B variants apply YaRN-style temperature scaling when extending to 128k inference.
3TogetherAI's blog post on long-context extension explicitly documents the attention-temperature factor and its tuning.
4vLLM and SGLang both expose attention temperature scaling as a serving-time knob for models that need it.
5Llama 4 Maverick and Mistral Large 3 train natively at long context with the scaling baked in, sidestepping the need for inference-time temperature.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does attention temperature interact with RoPE position interpolation (PI) and YaRN?
Sign in or go Pro to see the approach for this follow-up.
QWhy does attention magnitude grow with sequence length even when Q and K vectors stay bounded?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens if you apply attention temperature scaling at training time instead of inference time?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Inverting the direction. Dividing by greater than 1 softens; multiplying by greater than 1 sharpens. Getting this backward will make over-peaked attention worse, not better.