What is the primary reason for running LLM judges at temperature=0 in an eval pipeline? · Zenaique

Attempt it

Quick reference

TL;DR

Temperature=0 makes the judge greedy, so the same input yields the same verdict. That reproducibility lets you attribute eval score changes to your model, not to judge noise.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a judge who scores essays. If you hand the same essay back tomorrow and get a different grade, you can never tell whether the essay changed or the judge's mood did. Setting temperature=0 is like telling the judge to always make the same call on the same essay. Now if next week's score drops, you know the essay (your model's output) changed, not the grader. Without this, every eval run wobbles a little, and you waste hours chasing score changes that were just random. Note the judge is still not a perfect robot: hardware and batching can nudge results slightly, so equal scores are likely but not guaranteed.

Key concepts

LLM-as-judge Reproducibility Greedy decoding Regression testing

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: greedy decoding gives reproducibility, why that matters for regression CI, the diversity tradeoff, and the near-determinism caveat that means you also pin the model version.

Real-world examples

Real products, models, and research that use this idea.

LangSmith and Promptfoo default their LLM-judge evaluators to temperature=0 so regression comparisons across runs stay stable.
RAGAS runs its faithfulness and relevance judges with Claude Opus 4.7 or GPT-5.5 at temperature=0 for repeatable scores.
OpenAI Evals documents pinning both temperature and the judge model snapshot to keep eval results comparable over time.
Self-consistency judging deliberately samples a judge at temperature greater than 0 several times and majority-votes for a robust verdict.
DeepEval surfaces judge variance across repeated runs, which is the metric teams watch to confirm temperature=0 is actually pinned.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf temperature=0 makes the judge deterministic, why might two identical eval runs still disagree?

Sign in or go Pro to see the approach for this follow-up.

QWhen would you deliberately run the judge at temperature greater than 0?

Sign in or go Pro to see the approach for this follow-up.

QDoes temperature=0 do anything about position bias or self-preference bias?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming temperature=0 fixes judge bias. It only fixes sampling noise. Position bias and self-preference survive at any temperature and need separate mitigations.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.