When does temperature > 0 produce better results than temperature = 0 in production LLM applications?

**Temperature** is a single decoding knob that controls how the model picks its next token from the probability distribution. At T = 0 the model always picks the highest-probability token, which makes it deterministic. At T > 0 the model samples, so the same prompt can yield different outputs each run. The choice is not about which is better in the abstract. It is about what the task needs. Tasks with a single right answer (classification, JSON extraction, math) usually want T = 0 because reproducibility and consistency matter more than variety. Tasks that benefit from diversity (creative writing, brainstorming, generating multiple options for a UI) usually want T > 0 so the output is not stuck on one default phrasing. There is also a structural case for T > 0. Self-consistency voting samples many reasoning paths and votes on the most common answer. That pattern collapses at T = 0 because every sample is identical, so there is nothing to vote on.

Three distinct scenarios favor T > 0 over T = 0 in production. First, tasks that need genuine diversity, such as creative writing, brainstorming, or any UI that surfaces multiple options for the user to pick from. T = 0 forces every output toward the same modal answer, which kills variety. Second, sampling-based patterns like self-consistency. Self-consistency samples N reasoning paths and votes on the most-common final answer. At T = 0 all N samples are identical because greedy decoding is deterministic, so the vote collapses to one sample and the pattern stops working. Third, escaping a wrong argmax. Sometimes the model's top-1 token is incorrect even though good answers have meaningful probability mass nearby. T > 0 lets sampling find them; T = 0 locks in the wrong choice every run. Best-of-N over a verifier exploits this directly. T = 0 wins where the task has a single correct answer and you want reproducibility: classification heads, structured extraction, regression tests in your eval pipeline. The right framing is not which is universally better. It is which property the task needs, determinism or controlled diversity.