What is self-preference bias in LLM-as-judge setups?

**Self-preference bias** (also called self-enhancement bias) is when a model used as a judge gives higher scores to answers that look like its own writing style. It is one of the well-known weaknesses of using an LLM to grade other LLMs. Here is why it happens. A judge model learned what 'good writing' looks like from its own training data. So it quietly associates its own style (its phrasing, structure, level of detail) with quality. When two answers are equally good, the one that matches the judge's style gets a small unfair boost. The classic example: use GPT-4 as a judge to compare a GPT answer against a Claude answer. The GPT answer tends to score a little higher just because it sounds like what the judge would have written, not because it is actually better. This matters most when you compare models from different families head to head. The bias is usually small, but it can flip a close ranking and lead you to pick the wrong model. The simplest fix is to use a judge from a different model family than the answers you are scoring, or to use several judges and average their scores.

**Self-preference bias** is the tendency of an LLM judge to score outputs that match its own family's style higher than equally good outputs from other families. It is a systematic error, not random noise. The cause is simple. A judge learned what a good answer looks like from its training distribution. So it conflates 'matches my style' with 'is high quality'. A GPT judge tends to reward GPT-style answers; a Claude judge rewards Claude-style answers. Why it matters: the bias is small in absolute terms, but it skews head to head comparisons across model families. A same-family judge gives a systematically inflated answer. The main fix is to use a judge from a different family than the candidate. Two stronger options exist. First, ensemble several judges from different families and average or take the majority. Second, use an open-weight judge trained to be model-agnostic, like Prometheus 2. A cheap structural fix also helps. Force the judge to cite specific evidence from the answer before scoring. That anchors the score to content rather than to surface style.