How does the temperature parameter reshape the sampling distribution?

Attempt it

Temperature T divides the raw logits before softmax: p_i = softmax(logits_i / T). T < 1 sharpens the distribution by amplifying the gap between top logits, pushing probability mass toward the most likely tokens (more deterministic, more repetitive). T > 1 flattens it, raising the probability of low-logit tokens (more diverse, more random). T = 1 leaves the softmax untouched. T = 0 is the degenerate case the runtime treats as argmax (greedy decoding). It is a logit-domain rescaling, not a post-softmax operation, which is why the effect is non-linear in token probability.

Your answer

**Temperature** is a number you pass to an LLM that controls how random its output is. At every decoding step, the model produces a vector of raw scores called logits, one per possible next token. Normally those logits go through softmax to become a probability distribution, and the runtime samples one token from that distribution. Temperature divides the logits before the softmax. If temperature is less than 1, the gap between the largest logit and the rest gets amplified, so the top tokens get most of the probability mass. The output becomes more predictable and sometimes repetitive. If temperature is greater than 1, the gap shrinks, so even unlikely tokens get a meaningful chance of being picked. The output becomes more diverse and sometimes wilder. The two special values matter. Temperature equals 1 means no scaling at all; sampling uses the trained distribution as-is. Temperature equals 0 is the degenerate case where every probability except the top one would round to zero. Runtimes detect this and switch to greedy decoding: just pick the token with the highest logit, no randomness. Most chat APIs default to around 0.7 to 1.0, which keeps responses interesting without going off the rails. Code-generation use cases often run lower, around 0.0 to 0.3.

**Temperature** `T` is a scalar that scales the model's logits before softmax: `p_i = softmax(logits_i / T)`. The effect is to reshape the categorical distribution that decoding samples from, without changing the underlying model. The key insight is that this happens in logit space, not probability space, so the effect is non-linear. Dividing logits by `T < 1` amplifies the differences between large and small logits, because the exponential in softmax magnifies those gaps. Two tokens whose logits differ by 3 become much more separated after softmax with low temperature than two tokens whose logits differ by 0.3. The result is a distribution that puts almost all mass on the top few tokens, and the model behaves almost like a greedy decoder. Multiplying by `T > 1` compresses the gaps, pushing mass toward the tail of the distribution; the model becomes more willing to pick unusual tokens. Three special points anchor intuition: - `T = 1`: identity. The softmax is computed on the raw logits, sampling matches the trained distribution. - `T → 0`: the distribution collapses to a one-hot at the argmax. Runtimes treat `T = 0` as greedy decoding (just pick the highest logit, no random number drawn). - `T → ∞`: the distribution flattens to uniform; sampling becomes purely random across the vocabulary. In practice, chat APIs default to around `0.7` to `1.0` for general use. Code generation, summarization, and tasks where determinism matters often use `0.0` to `0.3`. Creative writing and brainstorming sometimes go above `1.0`. Temperature stacks with `top_p` and `top_k` truncation: temperature reshapes the distribution, then truncation cuts the tail, then sampling draws from what remains.

Quick reference

TL;DR

Temperature divides the logits before softmax; `T<1` sharpens toward the top token, `T>1` flattens toward the tail, `T=0` is greedy argmax.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the model's raw output as a row of heights, one per possible next word. The highest height is the model's favorite token. Temperature is a knob that scales those heights before they get turned into probabilities. Turn the knob below 1 and the gaps between heights grow, so the favorite towers over everyone else and almost always wins; the output gets predictable, sometimes repetitive. Turn it above 1 and the gaps shrink, so unlikely tokens get a real chance and the output gets more diverse, sometimes off the rails. Set it to 1 and nothing changes; you sample from the trained distribution as-is. Set it to 0 and you skip sampling entirely, just pick the tallest, every time.

Key concepts

Decoding Sampling Softmax Greedy Decoding

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: write the formula `p = softmax(logits / T)`, explain low / equal to one / high regimes, describe the T = 0 special case (greedy argmax), state that the effect is non-linear because softmax is non-linear, and give a typical default plus one production pin-low use case.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's Chat Completions API accepts `temperature` from 0.0 to 2.0 and defaults to 1.0; code-gen and structured-output use cases commonly set 0.0 to 0.2.
Anthropic's Messages API takes `temperature` from 0.0 to 1.0 with a default near 1.0, and tool-use prompts often pin it to 0 for deterministic argument selection.
vLLM, TGI, and SGLang all expose temperature as a sampling parameter that runs before `top_p` / `top_k` truncation in the engine.
Hugging Face Transformers' `generate()` exposes `do_sample=False` as an explicit greedy path that bypasses temperature entirely, equivalent to `T = 0` at the API level.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does temperature 0 not guarantee deterministic output on a GPU?

Sign in or go Pro to see the approach for this follow-up.

QHow does temperature interact with `top_p` truncation?

Sign in or go Pro to see the approach for this follow-up.

QWhy is the same temperature value not directly comparable across model families?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Describing temperature as scaling probabilities directly. It scales logits before softmax, which is non-linear in token probability and far more aggressive than a linear rescale.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.