What is self-preference bias in LLM-as-judge setups?

Attempt it

A model used as a judge tends to prefer outputs that are stylistically similar to its own training distribution. For example, using GPT-4 as judge unfairly favors GPT-4-generated answers over outputs from other model families, because the judge associates its own style with 'quality'. This undermines judge independence and inflates scores for outputs from the same model family as the judge.

Your answer

Quick reference

TL;DR

Self-preference bias is when an LLM judge scores outputs that match its own family's style higher than equally good outputs from other families. Fix it with a different judge model.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a teacher who learned to write in one particular style, and now they grade student essays. Without meaning to, they give higher marks to essays that sound like how they write, even when another essay is just as good. The teacher confuses 'looks like me' with 'is good'. LLM judges do the same thing. A model trained mostly on one style of answer learns to associate that style with quality. When you ask it to grade answers, it quietly rewards outputs that match its own habits and slightly penalizes the rest. The simplest fix is to bring in a different grader from a different family, so nobody is grading their own kind of work.

Key concepts

LLM-as-judge Self-preference bias Judge ensembling

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define self-preference bias, explain why it happens from the training distribution, why it skews cross-family comparisons, and the different-family or ensemble judge fix.

Real-world examples

Real products, models, and research that use this idea.

RAGAS lets you swap the judge model (Claude Opus 4.7 or GPT-5.5) so you avoid grading a family with its own model.
Prometheus 2 is the dominant open-weight judge in 2026, used precisely to sidestep commercial-family self-preference.
Anthropic Workbench judge templates explicitly warn about self-preference when scoring Claude outputs with a Claude judge.
Chatbot Arena uses crowd-sourced human votes rather than a single LLM judge, which avoids self-preference entirely.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you measure how much self-preference bias your judge actually has?

Sign in or go Pro to see the approach for this follow-up.

QWhy does ensembling judges from different families reduce self-preference more than running one judge many times?

Sign in or go Pro to see the approach for this follow-up.

QWhen would a single same-family judge still be acceptable?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using the same model family as both judge and candidate, then trusting the score. The judge inflates its own family's outputs, so the comparison is no longer fair.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.