Flashcard: what does the temperature parameter do during LLM generation and how should you set it?

**Temperature** is a number you pass to an LLM API (usually between 0 and 2) that controls how random or how predictable the model's output is. Under the hood, the model produces a score (called a logit) for every possible next token. Temperature divides those scores before the softmax that turns them into probabilities. Dividing by a small number (low temperature, like 0.2) makes the gap between the top score and the others bigger, so the most likely token gets almost all the probability and the model picks it. Dividing by a larger number (high temperature, like 1.0 or 1.2) shrinks the gaps so lower-ranked tokens have a real chance of being picked. The practical effect. Low temperature gives you safe, predictable, repeatable answers: good for extraction, classification, and structured output. High temperature gives you varied, creative, less predictable answers: good for brainstorming, fiction, or generating alternatives. Temperature 0 is a special case: it tells the model to always pick the single most likely next token, which is called greedy decoding.

**Temperature** is the scaling parameter applied to the model's output logits during sampling. The standard formula is softmax(logits / T). The temperature value reshapes the next-token probability distribution before a token is drawn from it. T < 1 (typically 0.1 to 0.7) sharpens the distribution toward the highest-probability tokens. The most likely token gets even more probability, lower-ranked tokens get squeezed. The result is more deterministic, more predictable output. T = 0 is the limit case: the distribution collapses to a one-hot on the argmax, which is greedy decoding. T > 1 (typically 1.0 to 1.5 in practice) flattens the distribution so lower-probability tokens have a real chance of being sampled. The result is more diverse, more creative output, with a higher risk of incoherence or off-topic generation as T grows. The practical mapping. Use low T (0-0.3) for factual extraction, classification, code generation, structured outputs where you want the same answer every time on the same input. Use medium T (0.5-0.8) for general assistant tasks where some variety is acceptable but coherence still matters. Use high T (0.9-1.2) for brainstorming, creative writing, generating multiple alternatives, or driving exploration in self-consistency loops. One caveat: T = 0 is not strictly deterministic in production. Sampling implementation, distributed inference, and ties in logits can still produce different outputs on the same input. For full determinism you also need a fixed seed plus a deterministic inference path.