How does the temperature parameter reshape the sampling distribution?

Flashcard·Easy·4.0 · 0·~30s·Asked atDecagon Promptlayer·Relevant atOpenAI

Attempt it

Temperature T divides the raw logits before softmax: p_i = softmax(logits_i / T). T < 1 sharpens the distribution by amplifying the gap between top logits, pushing probability mass toward the most likely tokens (more deterministic, more repetitive). T > 1 flattens it, raising the probability of low-logit tokens (more diverse, more random). T = 1 leaves the softmax untouched. T = 0 is the degenerate case the runtime treats as argmax (greedy decoding). It is a logit-domain rescaling, not a post-softmax operation, which is why the effect is non-linear in token probability.

Your answer

**Temperature** is a number you pass to an LLM that controls how random its output is. At every decoding step, the model produces a vector of raw scores called logits, one per possible next token. Normally those logits go through softmax to become a probability distribution, and the runtime samples one token from that distribution. Temperature divides the logits before the softmax. If temperature is less than 1, the gap between the largest logit and the rest gets amplified, so the top tokens get most of the probability mass. The output becomes more predictable and sometimes repetitive. If temperature is greater than 1, the gap shrinks, so even unlikely tokens get a meaningful chance of being picked. The output becomes more diverse and sometimes wilder. The two special values matter. Temperature equals 1 means no scaling at all; sampling uses the trained distribution as-is. Temperature equals 0 is the degenerate case where every probability except the top one would round to zero. Runtimes detect this and switch to greedy decoding: just pick the token with the highest logit, no randomness. Most chat APIs default to around 0.7 to 1.0, which keeps responses interesting without going off the rails. Code-generation use cases often run lower, around 0.0 to 0.3.

**Temperature** `T` is a scalar that scales the model's logits before softmax: `p_i = softmax(logits_i / T)`. The effect is to reshape the categorical distribution that decoding samples from, without changing the underlying model. The key insight is that this happens in logit space, not probability space, so the effect is non-linear. Dividing logits by `T < 1` amplifies the differences between large and small logits, because the exponential in softmax magnifies those gaps. Two tokens whose logits differ by 3 become much more separated after softmax with low temperature than two tokens whose logits differ by 0.3. The result is a distribution that puts almost all mass on the top few tokens, and the model behaves almost like a greedy decoder. Multiplying by `T > 1` compresses the gaps, pushing mass toward the tail of the distribution; the model becomes more willing to pick unusual tokens. Three special points anchor intuition: - `T = 1`: identity. The softmax is computed on the raw logits, sampling matches the trained distribution. - `T → 0`: the distribution collapses to a one-hot at the argmax. Runtimes treat `T = 0` as greedy decoding (just pick the highest logit, no random number drawn). - `T → ∞`: the distribution flattens to uniform; sampling becomes purely random across the vocabulary. In practice, chat APIs default to around `0.7` to `1.0` for general use. Code generation, summarization, and tasks where determinism matters often use `0.0` to `0.3`. Creative writing and brainstorming sometimes go above `1.0`. Temperature stacks with `top_p` and `top_k` truncation: temperature reshapes the distribution, then truncation cuts the tail, then sampling draws from what remains.

TL;DR

Temperature divides the logits before softmax; `T<1` sharpens toward the top token, `T>1` flattens toward the tail, `T=0` is greedy argmax.

Memory aid

Easy to grasp

Imagine the model's raw output as a row of heights, one per possible next word. The highest height is the model's favorite token. Temperature is a knob that scales those heights before they get turned into probabilities. Turn the knob below 1 and the gaps between heights grow, so the favorite towers over everyone else and almost always wins; the output gets predictable, sometimes repetitive. Turn it above 1 and the gaps shrink, so unlikely tokens get a real chance and the output gets more diverse, sometimes off the rails. Set it to 1 and nothing changes; you sample from the trained distribution as-is. Set it to 0 and you skip sampling entirely, just pick the tallest, every time.

Key concepts

Decoding Sampling Softmax Greedy Decoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Temperature is the simplest sampling knob and the one that most often gets misunderstood. Engineers talk about it as if it controls creativity or randomness in some abstract sense; under the hood it is a single scalar that divides the logits before softmax. That mechanical fact carries surprising power, because softmax is non-linear, and small temperature changes produce large probability changes once you cross certain thresholds.

This deep dive starts with the math, walks through what happens at each interesting value, covers the three special cases (T = 1, T = 0, T → ∞), explains how temperature stacks with the other sampling parameters, and ends with production realities like determinism and cross-model calibration.

The math: logits in, distribution out

The forward pass of an LLM produces a vector of unnormalized scores called logits, one per token in the vocabulary. The vocabulary is large, typically 50k to 200k tokens. To sample the next token, the decoder turns these logits into a probability distribution via softmax, then draws a token from that distribution.

The canonical softmax is p_i = exp(logits_i) / Σ_j exp(logits_j). Temperature inserts a scaling step before softmax:

p_i = \frac{\exp(\mathrm{logits}_i / T)}{\sum_j \exp(\mathrm{logits}_j / T)}

Dividing by T < 1 amplifies the logits, so the largest one looms even larger after exponentiation. Dividing by T > 1 shrinks them, so the gap between large and small narrows. The crucial property is that the rank order of tokens does not change; only the probability mass each one receives changes.

It matters that this is logit-space scaling, not probability-space rescaling. If you tried to sharpen the distribution by raising every probability to a power and renormalizing, you would get a similar shape but the calibration would differ from what temperature produces. Temperature is the operation that respects the geometry of the model's confidence; doing the math in logit space is what the framework expects.

Walking the temperature axis

Two special cases: zero and one

Stacking with top_p, top_k, and the random seed

Production realities: defaults, calibration, and cost

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's Chat Completions API accepts `temperature` from 0.0 to 2.0 and defaults to 1.0; code-gen and structured-output use cases commonly set 0.0 to 0.2.
Anthropic's Messages API takes `temperature` from 0.0 to 1.0 with a default near 1.0, and tool-use prompts often pin it to 0 for deterministic argument selection.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does temperature 0 not guarantee deterministic output on a GPU?

Argmax over floating-point logits is deterministic only if the logits themselves are bit-identical run to run. GPU non-determinism in fused attention kernels, atomic reductions in matmul, and library version drift can perturb logits at the LSB level, occasionally flipping the argmax on near-ties.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Describing temperature as scaling probabilities directly. It scales logits before softmax, which is non-linear in token probability and far more aggressive than a linear rescale.

60 second bullets to scan on the way to the call.

Write the temperature formula: divide logits by T before softmax.
Explain what happens at T < 1, T = 1, and T > 1.

Primary sources. Browse if you want the original framing.