What is LLM-as-judge and why did it emerge as an evaluation approach?

Attempt it

Your answer

Quick reference

TL;DR

LLM-as-judge uses a strong model to score other models' outputs against a rubric. It scales evaluation cheaply where human review is slow and string metrics miss meaning.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine grading thousands of essays. You could hire human graders, but they are slow and expensive. You could check whether each essay contains exact keywords, but that misses essays that say the right thing in different words. So instead you hire one very smart reader and give it a clear rubric: "Is the answer helpful? Is it accurate? Is the tone right?" That smart reader can read thousands of essays an hour, understand what each one means rather than just matching words, and apply the same rubric to every one. That smart reader is an LLM-as-judge. The catch is that even a smart reader has habits and blind spots, so you cannot trust it blindly. But for grading at scale, it beats both the human graders and the keyword checker.

Key concepts

LLM-as-judge Evaluation at scale Judge calibration

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define LLM-as-judge, contrast with human eval and string metrics, name what it captures that they cannot, and close on the proxy caveat plus calibration.

Real-world examples

Real products, models, and research that use this idea.

RAGAS uses a strong model (Claude Opus 4.7 or GPT-5.5) as judge to score RAG faithfulness and answer relevance at scale.
Chatbot Arena ranks frontier models using crowd preferences, and LLM judges are widely used to approximate that signal cheaply offline.
LangSmith and Promptfoo ship LLM-judge evaluators as a default for grading open-ended outputs in CI pipelines.
OpenAI Evals lets teams define model-graded checks where a judge model scores candidate completions against a rubric.
Prometheus 2 is a popular open-weight judge in 2026, used when teams want a model-agnostic grader to avoid commercial self-preference.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do lexical metrics like BLEU and ROUGE fail for open-ended LLM outputs?

Sign in or go Pro to see the approach for this follow-up.

QHow would you check that an LLM judge actually agrees with human raters?

Sign in or go Pro to see the approach for this follow-up.

QWhen is a judge the wrong tool and you should use exact match instead?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating the judge's score as ground truth. It is a scalable proxy with known biases that needs periodic calibration against human labels, not an oracle.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.