Dividing pre-softmax attention scores by an extra factor > 1 at inference does what?

MCQ·Medium·4.0 · 0·~1 min·Asked atDatabricks Ola Perplexity·Relevant atAnthropic

Attempt it

Softmax converts raw scores into a probability distribution. Its behavior is sensitive to the magnitude of the input scores: large differences between scores produce a peaked distribution (almost all mass on the top score), small differences produce a flat distribution (mass spread across many scores). Dividing all scores by a number greater than 1 before softmax shrinks the differences between them. The top score is still the largest, but the gap to the runner-ups is smaller. After softmax, the resulting attention weights are flatter, more mass goes to keys that previously got little weight. This is sometimes called attention temperature greater than 1 (by analogy to sampling temperature in language model decoding). It's used in long-context inference because some models, when stretched to context lengths well beyond what they were trained on, start showing attention patterns where almost all the weight collapses onto the few most recent tokens. Distant content is effectively ignored. Softening the attention distribution at inference time, without retraining, can recover some of that distant attention. The opposite move (multiplying scores by greater than 1, or dividing by less than 1) sharpens the distribution, making it more peaked. That's rarely useful at inference because trained attention is usually already peaked enough; sharpening tends to make the model brittle.

**The direction matters.** Dividing pre-softmax scores by a factor greater than 1 softens attention. Multiplying (or dividing by less than 1) sharpens it. The mechanism is simple. Softmax exponentiates differences in input scores. If the top score beats the next-best by 5 units, softmax assigns roughly e^5 = 148 times more weight to the top. If you divide all scores by 2, the gap becomes 2.5 units, the weight ratio drops to about 12x. The peak is still there but flatter. **Why this comes up in long-context inference.** Models trained at shorter context (say 4k tokens) sometimes show attention collapse when run at 32k+ tokens. Attention concentrates on the last few hundred tokens; distant content in the prompt gets near-zero weight. The standard explanation is that the dot-product magnitudes scale with sequence length while the softmax temperature stays fixed, so the effective distribution becomes too peaked. Dividing the scores by a length-dependent factor counteracts this. Production usage shows up in length-extension recipes. **YaRN** (Yet Another RoPE extensioN), used by Mistral and Llama variants, includes a temperature term that softens attention as a function of context length. The Mistral team's NTK-aware scaling does something similar. The values are typically small, 1.05 to 1.5, applied during long-context inference only. **The risk is over-softening.** Push the factor too high and attention becomes near-uniform, which destroys the model's ability to focus on the relevant tokens. The right value is tuned empirically per model and context length.

TL;DR

Dividing pre-softmax logits by a factor greater than 1 flattens the attention distribution, useful for redistributing weight away from over-peaked recent tokens at long context.

Memory aid

Easy to grasp

Picture a roomful of people voting on what to do next. If everyone's votes are very lopsided, the loudest few decide everything. Dividing every vote count by a number bigger than 1 before tallying brings everyone's totals closer together, so quieter voters get more say. The model works the same way when it picks which earlier words to focus on. Shrinking all the raw scores closer together before the final tally spreads its focus across distant words instead of pinning everything to the few loudest recent ones. That is exactly what you want when a long-prompt model has started ignoring useful information far back in the conversation.

Key concepts

Softmax function Long-context inference Temperature scaling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Attention temperature scaling is a small but consequential knob in modern long-context inference. The setup is mundane: divide the pre-softmax attention scores by a factor greater than 1 to soften the resulting distribution. The motivation is less obvious: as you stretch a transformer to context lengths far beyond what it was trained on, attention often collapses onto the most recent tokens, ignoring distant content that the model should be using. Temperature scaling at inference time can rebalance the distribution back.

The technique is part of the YaRN length-extension recipe and shows up in production Llama 3 long-context variants, Mistral extensions, and other community-tuned long-context models. It works because softmax is fundamentally sensitive to the magnitude of its inputs, and the relationship between attention magnitude and sequence length is not constant.

This deep dive walks through the softmax math, the empirical observation that motivates the technique, the YaRN-style formula in detail, and the trade-offs that determine when it helps versus hurts.

Softmax's sensitivity to logit magnitude

Softmax is shift-invariant: adding a constant to every input leaves the output unchanged. But it is not scale-invariant: multiplying every input by c > 1 makes the output more peaked, and dividing by c > 1 makes it flatter.

The math

For inputs (s_1, s_2, ..., s_n), softmax produces:

\text{softmax}(s)_i = \frac{e^{s_i}}{\sum_j e^{s_j}}

Dividing each s_i by t > 1 before softmax gives:

\text{softmax}(s/t)_i = \frac{e^{s_i / t}}{\sum_j e^{s_j / t}}

As t increases, the differences between scores shrink toward zero, and the distribution approaches uniform (1/n for every i). As t decreases below 1, the differences grow, and the distribution approaches a one-hot at the argmax.

Intuition

Think of softmax as 'soft argmax'. The temperature controls how soft. At t = 1 you get standard attention; at t > 1 it spreads weight more evenly; at t = inf it becomes uniform; at t -> 0 it collapses to one-hot.

This is the same temperature parameter used in language model sampling, just applied in a different part of the model. Sampling temperature scales the next-token logits; attention temperature scales the attention scores.

Why long context causes attention collapse

The YaRN-style scaling recipe

Limits, risks, and when it's the wrong fix

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

YaRN (Yet Another RoPE extensioN) by Peng et al. 2023 uses a temperature term in its length-extension recipe and is integrated into Mistral and Llama variants.
Together AI's long-context Llama 3 70B variants apply YaRN-style temperature scaling when extending to 128k inference.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does attention temperature interact with RoPE position interpolation (PI) and YaRN?

PI rescales RoPE angles to map a longer inference context onto the trained position range; YaRN extends PI with per-dimension scaling and a magnitude correction. Attention temperature is a separate knob that corrects softmax magnitude. The full YaRN recipe applies both: PI-like position scaling plus a sqrt(t)-scaled attention factor. Using one without the other typically underperforms either alone.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Inverting the direction. Dividing by greater than 1 softens; multiplying by greater than 1 sharpens. Getting this backward will make over-peaked attention worse, not better.

60 second bullets to scan on the way to the call.

Direction of effect: dividing scores by greater than 1 softens, multiplying sharpens
Why softmax becomes more peaked when logit magnitudes grow

Primary sources. Browse if you want the original framing.