Click any words you think contain an error. Click again to unmark.
Temperature divides logits before softmax: low T sharpens (deterministic), T=1 leaves the distribution unchanged, high T flattens (more random). The passage inverts every claim.
Imagine a bag of marbles, one for each word the model could say next. Some marbles are bigger (the model likes those words more) and some are smaller. The model reaches in without looking and pulls one out. Bigger marbles get picked more often. Temperature is a magic dial. Turn it down low, and the big marbles grow huge while the small ones shrink to almost nothing. You almost always pull the same big marble, so the model says the safe, predictable word every time. Turn it up high, and all the marbles become roughly the same size. Now any word can come out, so the model gets creative and surprising, but sometimes weird. Set it in the middle, and the bag stays exactly as the model packed it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Temperature is the most common sampling parameter in LLM APIs and the most commonly misexplained. Candidates who have set T on the OpenAI playground often have an intuition that gets the direction right but cannot ground it in the softmax math. The passage in this question inverts every claim, which makes the spot-error diagnostic crisp.
The stakes are higher than the simplicity of the formula suggests. A misunderstanding of temperature direction has shipped production bugs at major chatbot companies: teams that wanted deterministic JSON output set T=1 thinking it was 'standard', then watched the schema break on roughly five percent of requests. Teams that wanted creative brainstorming set T=0 thinking 'higher' was wrong, then complained that the model gave the same answer every time. The interview value of getting this question right is exactly that it identifies whether the candidate has actually read the softmax math, not just touched the playground.
This deep dive walks the formula, the limits at T=0 and large T, the interaction with top-p and top-k, and the operational implications for code generation, creative writing, and structured output. By the end you should be able to pick a temperature for any task and justify it from the math.
The formula and the direction
The model produces logits z, one per vocabulary token. Softmax turns z into probabilities P:
The division by T happens BEFORE the exponential. When T is small (say 0.1), dividing magnifies the differences between logits. The largest logit dominates the softmax, and the resulting distribution concentrates mass on that one token. The sampler nearly always picks the argmax.
When T is large (say 2.0), the divisions shrink the gaps. After exponential and normalisation, all tokens get a meaningful share of probability. The sampler picks more diverse tokens.
When T equals 1, dividing by 1 changes nothing. The softmax sees the raw logits, and you sample from the model's native, calibrated distribution. This is the default for nearly every chat API.
The passage in the question inverts all three: T=0 is wrongly called creative (it is greedy), T=1 is wrongly called deterministic (it is unchanged), and low T is wrongly recommended for unpredictable outputs (it is for stable outputs).
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI API defaults to T=1 for chat completions; cookbook recipes for code generation explicitly drop to T=0 or 0.2.
- Anthropic Messages API documentation recommends T=0 for deterministic tasks and notes that Claude's calibrated distribution is at T=1.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat's the difference between T=0 and 'greedy decoding'?
They are equivalent in practice. T=0 is a limit (division by zero), implemented as argmax. Most frameworks short-circuit T=0 to skip the softmax entirely and just pick the top logit.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Inverting the direction: claiming low temperature is creative and high temperature is deterministic. The opposite is true, low T sharpens the softmax toward argmax.
60 second bullets to scan on the way to the call.
The temperature formula: divide logits by T before softmax
Direction: low T sharpens, high T flattens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.