When is LLM-as-judge an appropriate evaluation method for prompt outputs and what are its known biases that you need to mitigate?

LLM-as-judge is using a strong language model to score the outputs of another model (or the same model under different prompts). The fit is tasks where you cannot write a precise rubric: is this answer helpful, is this summary faithful, is this response well-reasoned. Hand-written rubrics work for math grading; they fail for nuanced quality judgments. The approach has three well-documented biases. Positional bias means that when the judge sees two responses, it tends to favor the one presented first by about 10-15 points. Verbosity bias means it tends to score longer responses higher even when content quality is the same. Self-preference bias means a judge from one model family tends to favor outputs from the same family. The mitigations are all known and cheap. Shuffle the order of options across calls and average. Calibrate the judge against a small human-labeled set. Use a stronger model as judge than the model being judged. Once these are in place, LLM-as-judge often matches or beats human eval on consistency, because humans disagree with each other too.

LLM as judge fits the gap between rubrics and humans. Use it when the quality signal is real but hard to write down: helpfulness, faithfulness, reasoning quality, factuality on nuanced claims. Rubric-based scoring works when the answer is checkable. LLM as judge works when only judgment will do, and human eval is too slow or expensive to do at scale. Three biases are well documented. Positional bias gives the first-presented option a 10-15 point advantage in pairwise scoring. Verbosity bias gives longer responses higher scores even when content is comparable. Self-preference bias means a judge from a given model family slightly favors outputs from the same family. The mitigations are well-rehearsed. Shuffle option order across calls and average pairs of scores to wash out position. Cap response length in the prompt being judged, or normalize for length post-hoc, to control verbosity. Use a stronger judge model than the model being evaluated, and where possible use a judge from a different family for cross-family comparisons. Calibrate against a small human-labeled gold set. Report agreement rate alongside scores. The production pattern is a judge prompt with explicit criteria, two-pass pairwise scoring with shuffled order, and a calibration audit every model upgrade. Run that and LLM as judge becomes a reliable backbone for prompt regression CI.