LLM-as-Judge
Also known as: LLM as judge, Model-graded eval
Use a stronger LLM to grade another LLM's outputs against a rubric. Fast, cheap, biased.
Using a powerful LLM (often GPT-4 or Claude) to score or rank outputs from another LLM along specified rubrics. Faster and cheaper than human eval; standard for regression testing LLM apps.
In practice
The dominant eval pattern for production LLM apps. Senior interviews probe rubric design, position bias, and calibration with human eval.
How it compares
LLM-as-judge is one specific eval technique; evaluation is the umbrella practice.
Related topics
Questions that mention this term
- Why string match accuracy fails as an LLM evaluation metric
- Running LLM-as-judge on your…
- Why is scoring only…
- LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.
- Explain the difference between pointwise and pairwise evaluation in one breath.
- Design the post-FT eval suite for a customer support fine-tune
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.