What is the primary reason for running LLM judges at temperature=0 in an eval pipeline?

An **LLM-as-judge** uses a strong language model to score the outputs of another model. Like any model, the judge samples its response token by token, and the temperature setting controls how random that sampling is. At the default temperature, the judge can give slightly different verdicts each time you run it on the same input. One run says 7 out of 10, the next says 8. That noise is poison for evaluation, because you want to compare scores across runs and trust the difference. Setting temperature=0 makes the judge greedy: it always picks the highest-probability token. The same prompt now produces the same output. This is reproducibility, and it is the correct answer here. Why does it matter? In regression testing you run the eval this week and last week and compare. If the judge wobbles on its own, you cannot tell whether your model improved or the judge just rolled different dice. Pinning temperature=0 removes that confound, so a score change points to your model. The other options are distractors. Temperature does not fix position bias, does not reliably sharpen sensitivity, and has nothing to do with the context window.

**Temperature=0** makes the judge decode greedily. The judge always picks the highest-probability token. So the same input produces the same verdict. That property is reproducibility. It is the whole point in an eval pipeline. You run the eval this week and last week and compare the scores. If the judge is stochastic, scores drift run to run even on identical inputs. You can no longer attribute a score change to your model versus judge noise. This matters more for a judge than for a generator. A generator's job is to produce varied, creative text, so sampling diversity helps. A judge's job is to be a stable measuring instrument. You want the ruler to read the same length twice. The distractors fail. Temperature does not target position bias. That bias survives at any temperature and needs order-swapping instead. Lower temperature does not reliably make scores more discriminative. And temperature has zero connection to context window limits. One honest caveat. Temperature=0 is not perfectly deterministic in practice. Floating-point non-associativity, batching, and backend changes can still shift a token. So pin the model version too, and expect near-determinism rather than a guarantee.