LLM-as-Judge
Also known as: LLM as judge, Model-graded eval
Use a stronger LLM to grade another LLM's outputs against a rubric. Fast, cheap, biased.
Using a powerful LLM (often GPT-4 or Claude) to score or rank outputs from another LLM along specified rubrics. Faster and cheaper than human eval; standard for regression testing LLM apps.
In practice
The dominant eval pattern for production LLM apps. Senior interviews probe rubric design, position bias, and calibration with human eval.
How it compares
LLM-as-judge is one specific eval technique; evaluation is the umbrella practice.
Related topics
Questions that mention this term
Related terms
Hallucination
When a model confidently makes up something that isn't true.
LLM Evaluation
Measuring whether an LLM does what you want, beyond "looks fine to me".
Perplexity
Exp(average cross-entropy) on held-out text; lower means the model is less surprised by real data.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
MMLU (Massive Multitask Language Understanding)
Multiple-choice benchmark across 57 academic subjects; the standard "raw knowledge" headline number.