How does the temperature parameter reshape the sampling distribution?

**Temperature** is a number you pass to an LLM that controls how random its output is. At every decoding step, the model produces a vector of raw scores called logits, one per possible next token. Normally those logits go through softmax to become a probability distribution, and the runtime samples one token from that distribution. Temperature divides the logits before the softmax. If temperature is less than 1, the gap between the largest logit and the rest gets amplified, so the top tokens get most of the probability mass. The output becomes more predictable and sometimes repetitive. If temperature is greater than 1, the gap shrinks, so even unlikely tokens get a meaningful chance of being picked. The output becomes more diverse and sometimes wilder. The two special values matter. Temperature equals 1 means no scaling at all; sampling uses the trained distribution as-is. Temperature equals 0 is the degenerate case where every probability except the top one would round to zero. Runtimes detect this and switch to greedy decoding: just pick the token with the highest logit, no randomness. Most chat APIs default to around 0.7 to 1.0, which keeps responses interesting without going off the rails. Code-generation use cases often run lower, around 0.0 to 0.3.

**Temperature** `T` is a scalar that scales the model's logits before softmax: `p_i = softmax(logits_i / T)`. The effect is to reshape the categorical distribution that decoding samples from, without changing the underlying model. The key insight is that this happens in logit space, not probability space, so the effect is non-linear. Dividing logits by `T < 1` amplifies the differences between large and small logits, because the exponential in softmax magnifies those gaps. Two tokens whose logits differ by 3 become much more separated after softmax with low temperature than two tokens whose logits differ by 0.3. The result is a distribution that puts almost all mass on the top few tokens, and the model behaves almost like a greedy decoder. Multiplying by `T > 1` compresses the gaps, pushing mass toward the tail of the distribution; the model becomes more willing to pick unusual tokens. Three special points anchor intuition: - `T = 1`: identity. The softmax is computed on the raw logits, sampling matches the trained distribution. - `T → 0`: the distribution collapses to a one-hot at the argmax. Runtimes treat `T = 0` as greedy decoding (just pick the highest logit, no random number drawn). - `T → ∞`: the distribution flattens to uniform; sampling becomes purely random across the vocabulary. In practice, chat APIs default to around `0.7` to `1.0` for general use. Code generation, summarization, and tasks where determinism matters often use `0.0` to `0.3`. Creative writing and brainstorming sometimes go above `1.0`. Temperature stacks with `top_p` and `top_k` truncation: temperature reshapes the distribution, then truncation cuts the tail, then sampling draws from what remains.