Flashcard: what does the temperature parameter do during LLM generation and how should you set it? · Zenaique

Attempt it

Temperature is a scalar that scales the model's output logits before the softmax during sampling. Mathematically, softmax(logits / T) controls how peaked the next-token distribution is. T < 1 sharpens the distribution toward the most likely tokens (more deterministic, less diverse). T > 1 flattens it (more variety, higher creativity, higher risk of incoherence). T = 0 collapses to greedy decoding: always pick the argmax token. Use low T (0-0.3) for factual extraction, classification, structured output; use higher T (0.7-1.0) for brainstorming, creative writing, generating diverse alternatives.

Your answer

**Temperature** is a number you pass to an LLM API (usually between 0 and 2) that controls how random or how predictable the model's output is. Under the hood, the model produces a score (called a logit) for every possible next token. Temperature divides those scores before the softmax that turns them into probabilities. Dividing by a small number (low temperature, like 0.2) makes the gap between the top score and the others bigger, so the most likely token gets almost all the probability and the model picks it. Dividing by a larger number (high temperature, like 1.0 or 1.2) shrinks the gaps so lower-ranked tokens have a real chance of being picked. The practical effect. Low temperature gives you safe, predictable, repeatable answers: good for extraction, classification, and structured output. High temperature gives you varied, creative, less predictable answers: good for brainstorming, fiction, or generating alternatives. Temperature 0 is a special case: it tells the model to always pick the single most likely next token, which is called greedy decoding.

**Temperature** is the scaling parameter applied to the model's output logits during sampling. The standard formula is softmax(logits / T). The temperature value reshapes the next-token probability distribution before a token is drawn from it. T < 1 (typically 0.1 to 0.7) sharpens the distribution toward the highest-probability tokens. The most likely token gets even more probability, lower-ranked tokens get squeezed. The result is more deterministic, more predictable output. T = 0 is the limit case: the distribution collapses to a one-hot on the argmax, which is greedy decoding. T > 1 (typically 1.0 to 1.5 in practice) flattens the distribution so lower-probability tokens have a real chance of being sampled. The result is more diverse, more creative output, with a higher risk of incoherence or off-topic generation as T grows. The practical mapping. Use low T (0-0.3) for factual extraction, classification, code generation, structured outputs where you want the same answer every time on the same input. Use medium T (0.5-0.8) for general assistant tasks where some variety is acceptable but coherence still matters. Use high T (0.9-1.2) for brainstorming, creative writing, generating multiple alternatives, or driving exploration in self-consistency loops. One caveat: T = 0 is not strictly deterministic in production. Sampling implementation, distributed inference, and ties in logits can still produce different outputs on the same input. For full determinism you also need a fixed seed plus a deterministic inference path.

Quick reference

TL;DR

Temperature scales the next-token logits before softmax; low T sharpens toward the most likely tokens (deterministic), high T flattens the distribution (diverse), and T = 0 forces greedy decoding.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine the model has a big spinner with one slot per possible next word, and the slot sizes are proportional to how likely each word is. Temperature is a knob that warps the spinner. Turning the knob down (low temperature) makes the most likely slot huge and the rest tiny: the spinner almost always lands on the obvious word. Turning the knob up (high temperature) flattens the slots so even unusual words have a real chance of being picked. Turn it all the way to zero and the spinner stops spinning; you just pick the biggest slot every time. Choose low temperature when you want safe, predictable answers like extracting a fact; choose higher temperature when you want creative variety like brainstorming.

Key concepts

Temperature (decoding)Sampling Top-p (nucleus sampling)

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define temperature as the softmax scaler, write the formula, map T values to tasks, distinguish from top-p, and call out the T = 0 determinism caveat.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's Chat Completions API exposes a temperature parameter 0-2 plus a separate seed parameter; GPT-5.5 in JSON-mode tasks typically runs at temperature 0 or 0.2 for predictable structured output.
Anthropic's Messages API for Claude Opus 4.7 accepts temperature 0-1, with prompt-engineering guides recommending 0 for extraction and 0.7-1.0 for creative writing.
Self-consistency techniques sample 5-20 reasoning chains at temperature 0.7-1.0 and majority-vote the answer; the diversity from temperature is what makes the vote informative.
vLLM and Ollama expose temperature alongside top-p and top-k as standard decoding parameters; common production defaults are T = 0.7 and top-p = 0.9 for general assistant tasks.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does temperature interact with top-p (nucleus) sampling?

Sign in or go Pro to see the approach for this follow-up.

QWhy might you want non-zero temperature for self-consistency even on a factual task?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the difference between temperature and repetition penalty?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Setting temperature to 0 and assuming the model is now fully deterministic across runs. Other sources of variance (sampling implementation, distributed inference, ties in logits) can still produce different outputs on identical inputs.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.