**Self-preference bias** is the documented effect that an LLM judge tends to score outputs from its own model family higher than equivalent outputs from a different family. It shows up in single-agent settings (Zheng et al., MT-Bench) and gets sharper in multi-agent because each agent persona is another surface where the judge can recognise its own style.
The bias is not about correctness on a math problem (which is checkable). It shows up on the open-ended quality dimensions: reasoning depth, coherence, helpfulness. The judge sees writing that feels familiar (sentence rhythm, paragraph shape, vocabulary choices) and rates it a little higher than blinded human raters would.
**Three mitigations that work.**
- **Cross-family judging.** Use a judge from a different model family than any agent in the system. Anthropic agents, OpenAI judge (or vice versa).
- **Judge ensembling.** Average scores across two or three different judge models so individual family preferences cancel.
- **Rubric scoring.** Constrain the judge to specific check items (did the agent call tool X with arg Y? did the critic flag the bug?) rather than open-ended 'how good was this'. Specific checks are harder to bias.
The three wrong answers fail for concrete reasons: there is measurable bias (well replicated), JSON failures are not the issue (the bias affects free-text scoring), and latency has nothing to do with self-preference.
**The risk is self-preference bias, amplified by multi-agent style surfaces.**
Self-preference is the well-documented effect where an LLM judge gives higher scores to outputs from its own model family than blinded human raters do. The original evidence (Zheng et al. on MT-Bench, follow-up work through 2024-2025) measured the gap at several points on Likert-style quality dimensions. The effect concentrates on subjective dimensions (reasoning depth, helpfulness, coherence) and largely disappears on objective dimensions (final-answer correctness on a math problem) because correctness is checkable independently.
**Why multi-agent makes it worse.** Each agent persona has its own writing style, and that style is shaped by the underlying model. A multi-agent trajectory from a single family carries the family's fingerprint across the planner, the executor, and the critic. The judge sees three style surfaces it recognises, not one, and the familiarity bias compounds. A cross-family trajectory exposes more 'foreign' style and gets penalised on the same subjective dimensions.
**The three mitigations.**
- **Cross-family judge.** If your agents are Claude family, judge with GPT-4o or Gemini. Eliminates the same-family effect at the cost of introducing the new judge's own biases (different shape, but bounded).
- **Judge ensemble.** Two or three judges from different families, averaged. Individual biases cancel; remaining noise is acceptable for most production decisions.
- **Rubric over Likert.** Replace 'rate this from 1 to 5 on helpfulness' with specific checks: did the planner produce a numbered plan, did the coder call `write_file`, did the critic raise an objection. Specific checks pin the judge to behaviour rather than style.
**Why the wrong answers fail.** 'No measurable bias' contradicts replicated empirical evidence. 'JSON failures' confuses output-format errors with judgment errors. 'Runs slower' is unrelated to bias and not even true.
Quick reference
TL;DR
Self-preference bias inflates judge scores when the judge shares a family with the agents; multi-agent amplifies it because each persona is another style surface.
Imagine a baking contest where the judge happens to be the head baker at one of the competing bakeries. Even if the judge tries to be fair, they will recognise their own style of icing, their own kind of crumb, their own preferred sweetness, and tend to score familiar cakes a little higher. The cakes are not better; they just feel right. Now imagine the contest has three courses per entry, so the judge sees their own style three times per competitor. The familiarity advantage compounds. That is what happens when an LLM judges multi-agent trajectories from its own model family.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Closed-form tasks with externally checkable answers (math, code that compiles, exact-match QA) Verification is independent of judgment, so an LLM judge from any family is fine; self-preference bias does not bind.
Cross-family judge when the alternative family is much weaker than the agent family A weaker judge may make basic errors that outweigh the bias-reduction benefit; pick a judge at or above the agent's capability tier.
Judge ensembling when latency or cost budgets cannot accommodate it Two or three judge calls per evaluated sample multiplies eval cost; reserve ensembles for high-stakes release gates.
Quick scan · 2–4 min
Everything important, quickly.
6 to 8 min: name the bias, cite the literature, explain why multi-agent amplifies it, walk through cross-family, ensembling, and rubric mitigations, name when each does not help.
Real-world examples
Real products, models, and research that use this idea.
1Zheng et al. (MT-Bench) measured a several-point self-preference bias for GPT-4 judging GPT-4 outputs versus blinded humans.
22026 production teams typically cross-judge Claude Opus 4.7 outputs with GPT-5.5 or Gemini 3.1 Pro on subjective dimensions to control for family bias.
3LangSmith and Braintrust let you configure ensembles of judge models for pairwise and Likert evaluations.
4The 2024 G-Eval and Prometheus 2 papers explicitly recommend rubric-based scoring to reduce open-ended judgment bias.
5Cohere's Aya and Mistral's evaluation work both use cross-family judges to publish multi-model benchmark results.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually measure self-preference bias in your own eval pipeline?
Sign in or go Pro to see the approach for this follow-up.
QWhen does cross-family judging not help much?
Sign in or go Pro to see the approach for this follow-up.
QHow does pairwise scoring change the bias profile?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing 'a model can be objective about its own outputs'. The literature is unambiguous: it cannot, and multi-agent makes it worse.