Flashcard: what does the temperature parameter do during LLM generation and how should you set it?

Flashcard·Easy·4.0 · 0·~30s·Asked atPolyai Robust Intelligence Vernacular Ai·Relevant atAnthropic

Attempt it

Temperature is a scalar that scales the model's output logits before the softmax during sampling. Mathematically, softmax(logits / T) controls how peaked the next-token distribution is. T < 1 sharpens the distribution toward the most likely tokens (more deterministic, less diverse). T > 1 flattens it (more variety, higher creativity, higher risk of incoherence). T = 0 collapses to greedy decoding: always pick the argmax token. Use low T (0-0.3) for factual extraction, classification, structured output; use higher T (0.7-1.0) for brainstorming, creative writing, generating diverse alternatives.

Your answer

**Temperature** is a number you pass to an LLM API (usually between 0 and 2) that controls how random or how predictable the model's output is. Under the hood, the model produces a score (called a logit) for every possible next token. Temperature divides those scores before the softmax that turns them into probabilities. Dividing by a small number (low temperature, like 0.2) makes the gap between the top score and the others bigger, so the most likely token gets almost all the probability and the model picks it. Dividing by a larger number (high temperature, like 1.0 or 1.2) shrinks the gaps so lower-ranked tokens have a real chance of being picked. The practical effect. Low temperature gives you safe, predictable, repeatable answers: good for extraction, classification, and structured output. High temperature gives you varied, creative, less predictable answers: good for brainstorming, fiction, or generating alternatives. Temperature 0 is a special case: it tells the model to always pick the single most likely next token, which is called greedy decoding.

**Temperature** is the scaling parameter applied to the model's output logits during sampling. The standard formula is softmax(logits / T). The temperature value reshapes the next-token probability distribution before a token is drawn from it. T < 1 (typically 0.1 to 0.7) sharpens the distribution toward the highest-probability tokens. The most likely token gets even more probability, lower-ranked tokens get squeezed. The result is more deterministic, more predictable output. T = 0 is the limit case: the distribution collapses to a one-hot on the argmax, which is greedy decoding. T > 1 (typically 1.0 to 1.5 in practice) flattens the distribution so lower-probability tokens have a real chance of being sampled. The result is more diverse, more creative output, with a higher risk of incoherence or off-topic generation as T grows. The practical mapping. Use low T (0-0.3) for factual extraction, classification, code generation, structured outputs where you want the same answer every time on the same input. Use medium T (0.5-0.8) for general assistant tasks where some variety is acceptable but coherence still matters. Use high T (0.9-1.2) for brainstorming, creative writing, generating multiple alternatives, or driving exploration in self-consistency loops. One caveat: T = 0 is not strictly deterministic in production. Sampling implementation, distributed inference, and ties in logits can still produce different outputs on the same input. For full determinism you also need a fixed seed plus a deterministic inference path.

TL;DR

Temperature scales the next-token logits before softmax; low T sharpens toward the most likely tokens (deterministic), high T flattens the distribution (diverse), and T = 0 forces greedy decoding.

Memory aid

Easy to grasp

Imagine the model has a big spinner with one slot per possible next word, and the slot sizes are proportional to how likely each word is. Temperature is a knob that warps the spinner. Turning the knob down (low temperature) makes the most likely slot huge and the rest tiny: the spinner almost always lands on the obvious word. Turning the knob up (high temperature) flattens the slots so even unusual words have a real chance of being picked. Turn it all the way to zero and the spinner stops spinning; you just pick the biggest slot every time. Choose low temperature when you want safe, predictable answers like extracting a fact; choose higher temperature when you want creative variety like brainstorming.

Key concepts

Temperature (decoding)Sampling Top-p (nucleus sampling)

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Temperature is the first decoding parameter most people meet and the one that gets misunderstood most often. The casual mental model ('higher equals more creative, lower equals more focused') is correct as far as it goes, but it does not tell you what the knob is actually doing inside the model, how it interacts with the other decoding parameters in a real stack, or where its limits sit.

The right mental model is concrete and mechanical. The model produces a logit (a real-valued score) for every possible next token; temperature scales those logits before the softmax that turns them into a probability distribution; a token is then sampled from the resulting distribution. Different temperatures shape the distribution differently, and that shape is what makes the output more focused or more diverse.

This deep dive walks through the math, the operational mapping from T values to task types, the interaction with top-p and top-k, and the caveats around T = 0 that surprise people the first time they need byte-deterministic replay in production.

The mechanics: what temperature does to the logits

At every generation step the model produces a logit vector z of size equal to the vocabulary (50k-200k tokens depending on tokenizer). To convert logits into a probability distribution, the model applies softmax. With temperature, softmax becomes:

p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

When T is small (close to 0), z_i / T becomes large in magnitude. The exponentials in the numerator and denominator stretch the gaps between high-scoring and low-scoring tokens, so the highest-scoring token captures almost all the probability mass and lower-scoring tokens get crushed toward zero. In the limit T -> 0, all the probability lands on the argmax token; this is greedy decoding.

When T is large (above 1), z_i / T shrinks in magnitude. The exponentials compress the gaps, so the distribution flattens. Lower-scoring tokens get a meaningful share of the probability mass, which means they have a real chance of being sampled. In the limit T -> infinity, the distribution approaches uniform over the vocabulary.

The sampling step then draws one token from the (possibly reshaped) distribution. Temperature does not affect what the model knows or what scores it produces; it only affects how those scores translate into a sampling distribution. This is why temperature is a decoding parameter, not a model parameter.

The mapping: which T value for which task

How temperature composes with top-p and top-k

The determinism caveat at T = 0

Temperature in self-consistency and ensemble patterns

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's Chat Completions API exposes a temperature parameter 0-2 plus a separate seed parameter; GPT-5.5 in JSON-mode tasks typically runs at temperature 0 or 0.2 for predictable structured output.
Anthropic's Messages API for Claude Opus 4.7 accepts temperature 0-1, with prompt-engineering guides recommending 0 for extraction and 0.7-1.0 for creative writing.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does temperature interact with top-p (nucleus) sampling?

Temperature scales the distribution; top-p truncates it to the smallest set whose cumulative probability exceeds p. They are usually applied together: temperature first, then top-p truncation, then sample.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Setting temperature to 0 and assuming the model is now fully deterministic across runs. Other sources of variance (sampling implementation, distributed inference, ties in logits) can still produce different outputs on identical inputs.

60 second bullets to scan on the way to the call.

Write the softmax formula with temperature; explain what T = 0 and T > 1 do
Map T values to task types (extraction vs creative vs assistant)

Primary sources. Browse if you want the original framing.