Self-preference bias is when an LLM judge scores outputs that match its own family's style higher than equally good outputs from other families. Fix it with a different judge model.
Imagine a teacher who learned to write in one particular style, and now they grade student essays. Without meaning to, they give higher marks to essays that sound like how they write, even when another essay is just as good. The teacher confuses 'looks like me' with 'is good'. LLM judges do the same thing. A model trained mostly on one style of answer learns to associate that style with quality. When you ask it to grade answers, it quietly rewards outputs that match its own habits and slightly penalizes the rest. The simplest fix is to bring in a different grader from a different family, so nobody is grading their own kind of work.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Self-preference bias (also called self-enhancement bias) is the tendency of an LLM judge to score outputs that resemble its own family's style higher than equally good outputs from other families. It is one of the headline weaknesses of the LLM-as-judge pattern, alongside position bias and weak factual grounding.
The reason this matters for an interview is that LLM-as-judge is now the default way teams evaluate generative systems at scale. Human evaluation is slow and expensive, so a single strong model grades thousands of candidate outputs per hour against a rubric. If you are going to trust a model to grade other models, you have to understand the systematic errors that grading introduces.
Self-preference is the one that most directly corrupts head to head comparisons between vendors, and it is the bias most people forget to control for. Position bias has an obvious mechanical fix (swap the order). Self-preference is subtler because it hides inside the judge's notion of quality itself. This deep dive walks through what the bias is, why it arises, when it actually hurts you, the practical fixes ordered by cost, and how to confirm the fix worked.
What the bias actually is
When you use one language model to score the outputs of another, the judge is supposed to rate quality: helpfulness, correctness, clarity, tone. Self-preference bias means the judge also quietly rewards a fourth thing it should ignore, namely how closely the answer matches its own style.
Concretely, give a GPT judge two answers of equal quality, one written in a GPT-like voice and one in a different family's voice. The GPT-style answer tends to score a few points higher. The judge is not lying. It genuinely perceives its own style as higher quality, because that is the style it was trained to produce as 'good'.
The effect is systematic, not random. It points the same direction every time you run it, which is exactly why you cannot get rid of it by averaging many runs of the same judge.
It is worth distinguishing self-preference from its neighbors. Position bias is about where an answer sits in the prompt. Length bias is about rewarding longer answers regardless of content. Self-preference is specifically about stylistic family resemblance: the judge favors the way it itself would have phrased the answer. These biases can stack, so a verbose, first-position, same-family answer collects three separate unearned advantages at once.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS lets you swap the judge model (Claude Opus 4.7 or GPT-5.5) so you avoid grading a family with its own model.
- Prometheus 2 is the dominant open-weight judge in 2026, used precisely to sidestep commercial-family self-preference.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you measure how much self-preference bias your judge actually has?
Score the same set of answers with a same-family judge and a different-family judge, then compare the per-family score gap. A consistent tilt toward one family signals the bias size.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using the same model family as both judge and candidate, then trusting the score. The judge inflates its own family's outputs, so the comparison is no longer fair.
60 second bullets to scan on the way to the call.
What self-preference bias is in one sentence
Why it happens (training distribution and style)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.