What is the primary reason for running LLM judges at temperature=0 in an eval pipeline?
Temperature=0 makes the judge greedy, so the same input yields the same verdict. That reproducibility lets you attribute eval score changes to your model, not to judge noise.
Imagine a judge who scores essays. If you hand the same essay back tomorrow and get a different grade, you can never tell whether the essay changed or the judge's mood did. Setting temperature=0 is like telling the judge to always make the same call on the same essay. Now if next week's score drops, you know the essay (your model's output) changed, not the grader. Without this, every eval run wobbles a little, and you waste hours chasing score changes that were just random. Note the judge is still not a perfect robot: hardware and batching can nudge results slightly, so equal scores are likely but not guaranteed.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Temperature=0 is the single most common configuration choice in an LLM-as-judge pipeline, and the reason is reproducibility. When a language model decodes, it samples the next token from a probability distribution over the vocabulary, and the temperature parameter rescales that distribution before sampling. Concretely, the model divides each logit by the temperature before the softmax. High temperature flattens the distribution and increases randomness; temperature=0 collapses it to argmax, picking the single highest-probability token every time. So the judge stops being a sampler and becomes a function: same input, same output.
The multiple-choice answer turns on knowing what that determinism buys you and, just as importantly, what it does not. The primary payoff is that you can compare eval scores across runs and trust the difference, which is the bedrock of any regression workflow. It is tempting to over-claim and say temperature=0 also fixes bias or sharpens the judge, and the distractors in this question are built from exactly those plausible-sounding over-claims.
The deep dive below works through why a judge needs reproducibility more than a generator does, how that reproducibility plugs into a regression CI gate, the diversity tradeoff you give up and the one case where you want it back, the three distractors and why each fails, and the honest caveat that temperature=0 is only near-deterministic rather than perfectly deterministic in real production systems.
Why a judge needs reproducibility more than a generator
A generator and a judge have opposite jobs. A generator should produce varied, fluent, sometimes creative text, so sampling diversity is a feature. You often run a generator at temperature 0.7 or higher precisely to avoid robotic, repetitive output, and you would never want every user to receive the identical phrasing.
A judge has the opposite job. It is a measuring instrument, and the first thing you demand of any instrument is that it reads the same value twice on the same object. If you weigh the same parcel and the scale says 2.0 kg, then 2.1 kg, then 1.9 kg, the scale is useless for tracking weight changes over time. The variation is not informative; it is just noise that drowns the signal you care about.
That is exactly the situation with a stochastic judge. You run your eval suite this week, get an aggregate score of 82, ship a model change, run it again next week, and get 80. Did the model regress, or did the judge just sample differently on a few borderline cases? With a stochastic judge you cannot tell, and the two-point drop is uninterpretable. Temperature=0 removes that ambiguity: the judge contributes zero sampling variance, so any score delta is attributable to your model, your prompt, or your test set rather than to the grader's mood.
This is why the framing matters. You are not asking the judge to be smart in some new way at temperature=0. You are asking it to be repeatable, which is a precondition for treating its scores as measurements at all rather than as one-off opinions.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangSmith and Promptfoo default their LLM-judge evaluators to temperature=0 so regression comparisons across runs stay stable.
- RAGAS runs its faithfulness and relevance judges with Claude Opus 4.7 or GPT-5.5 at temperature=0 for repeatable scores.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf temperature=0 makes the judge deterministic, why might two identical eval runs still disagree?
Point at floating-point non-associativity across GPU reductions, dynamic batching, and silent backend or model-version updates. Argmax can flip on a near-tie. The fix is pinning the model version and framework, plus logging a judge fingerprint per run.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming temperature=0 fixes judge bias. It only fixes sampling noise. Position bias and self-preference survive at any temperature and need separate mitigations.
60 second bullets to scan on the way to the call.
Why reproducibility is the primary reason for temperature=0
How greedy decoding makes the same input yield the same verdict
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.