T > 0 wins when you need diversity (creative tasks, self-consistency voting) or want to escape a wrong argmax; T = 0 wins for single-answer tasks and reproducibility.
Imagine asking a friend to pick a restaurant. If you always want their single best recommendation, you want them to give the same answer every time. That is temperature equals zero, totally consistent, no variety. But if you want them to brainstorm five different options for a group, you need them to sometimes pick the second or third favorite, not just the absolute top. That is temperature above zero, where the model is allowed to sample less obvious answers. The right setting depends on what the answer is for, not on a universal rule.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Temperature is the most-tuned and most-misunderstood decoding parameter in production LLM apps. Most teams set it once during prototyping and never revisit it, which is fine until a sampling-based pattern like self-consistency or best-of-N enters the codebase and silently breaks because someone left T = 0 in the config. The right framing treats temperature as a task-conditional choice, not a global default.
This question matters because it sits at the boundary between the determinism story (T = 0 for reproducibility) and the diversity story (T > 0 for creative and sampling-based patterns). Senior engineers do not pick one camp. They reason about what the task needs and pin T at the call site, so the eval suite and production traffic agree on which property is in play.
The deep dive walks through the mechanism, the three scenarios where T > 0 earns its keep, the scenarios where T = 0 wins, and how to wire the choice into the eval pipeline so it does not drift.
The mechanism, softmax with temperature
Temperature scales the logits before the softmax converts them into a probability distribution. The standard form is:
At T = 1 you get the raw model distribution. T below 1 sharpens it, concentrating mass on the top tokens. T above 1 flattens it, spreading mass into the tail. The interesting boundary case is T approaching 0, where the distribution becomes a one-hot vector on the argmax token. That is greedy decoding, and it is fully deterministic.
It is worth noticing what temperature cannot do. It cannot rewrite the logits z_i, only reshape them. The model's underlying belief about which tokens are good does not change with T. What changes is how aggressively the sampler commits to the top candidates versus exploring the alternatives.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's Claude Opus 4.7 API defaults T = 1 but production extraction pipelines pin T = 0 for reproducible JSON outputs.
- OpenAI's GPT-5.5 docs recommend T = 0 for classification and T = 0.7 to 1.0 for creative generation.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does self-consistency specifically need T > 0?
Voting requires distinct samples; at T = 0 all N samples are identical, so the vote collapses to a single sample.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating T = 0 as the safe default for every production task, then being surprised when self-consistency voting collapses because every sample is identical.
60 second bullets to scan on the way to the call.
Why T = 0 is deterministic (greedy argmax)
Three scenarios where T > 0 wins
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.