Identify the judge-bias risk when the same model judges its own multi-agent trajectories

MCQ·Hard·4.0 · 0·~1 min·Asked atAnthropic OpenAI Google

Attempt it

**Self-preference bias** is the documented effect that an LLM judge tends to score outputs from its own model family higher than equivalent outputs from a different family. It shows up in single-agent settings (Zheng et al., MT-Bench) and gets sharper in multi-agent because each agent persona is another surface where the judge can recognise its own style. The bias is not about correctness on a math problem (which is checkable). It shows up on the open-ended quality dimensions: reasoning depth, coherence, helpfulness. The judge sees writing that feels familiar (sentence rhythm, paragraph shape, vocabulary choices) and rates it a little higher than blinded human raters would. **Three mitigations that work.** - **Cross-family judging.** Use a judge from a different model family than any agent in the system. Anthropic agents, OpenAI judge (or vice versa). - **Judge ensembling.** Average scores across two or three different judge models so individual family preferences cancel. - **Rubric scoring.** Constrain the judge to specific check items (did the agent call tool X with arg Y? did the critic flag the bug?) rather than open-ended 'how good was this'. Specific checks are harder to bias. The three wrong answers fail for concrete reasons: there is measurable bias (well replicated), JSON failures are not the issue (the bias affects free-text scoring), and latency has nothing to do with self-preference.

**The risk is self-preference bias, amplified by multi-agent style surfaces.** Self-preference is the well-documented effect where an LLM judge gives higher scores to outputs from its own model family than blinded human raters do. The original evidence (Zheng et al. on MT-Bench, follow-up work through 2024-2025) measured the gap at several points on Likert-style quality dimensions. The effect concentrates on subjective dimensions (reasoning depth, helpfulness, coherence) and largely disappears on objective dimensions (final-answer correctness on a math problem) because correctness is checkable independently. **Why multi-agent makes it worse.** Each agent persona has its own writing style, and that style is shaped by the underlying model. A multi-agent trajectory from a single family carries the family's fingerprint across the planner, the executor, and the critic. The judge sees three style surfaces it recognises, not one, and the familiarity bias compounds. A cross-family trajectory exposes more 'foreign' style and gets penalised on the same subjective dimensions. **The three mitigations.** - **Cross-family judge.** If your agents are Claude family, judge with GPT-4o or Gemini. Eliminates the same-family effect at the cost of introducing the new judge's own biases (different shape, but bounded). - **Judge ensemble.** Two or three judges from different families, averaged. Individual biases cancel; remaining noise is acceptable for most production decisions. - **Rubric over Likert.** Replace 'rate this from 1 to 5 on helpfulness' with specific checks: did the planner produce a numbered plan, did the coder call `write_file`, did the critic raise an objection. Specific checks pin the judge to behaviour rather than style. **Why the wrong answers fail.** 'No measurable bias' contradicts replicated empirical evidence. 'JSON failures' confuses output-format errors with judgment errors. 'Runs slower' is unrelated to bias and not even true.

Quick reference

TL;DR

Self-preference bias inflates judge scores when the judge shares a family with the agents; multi-agent amplifies it because each persona is another style surface.

Memory aid

Easy to grasp

Imagine a baking contest where the judge happens to be the head baker at one of the competing bakeries. Even if the judge tries to be fair, they will recognise their own style of icing, their own kind of crumb, their own preferred sweetness, and tend to score familiar cakes a little higher. The cakes are not better; they just feel right. Now imagine the contest has three courses per entry, so the judge sees their own style three times per competitor. The familiarity advantage compounds. That is what happens when an LLM judges multi-agent trajectories from its own model family.

Key concepts

Multi-Agent Systems

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 to 8 min: name the bias, cite the literature, explain why multi-agent amplifies it, walk through cross-family, ensembling, and rubric mitigations, name when each does not help.

Real-world examples

Real products, models, and research that use this idea.

Zheng et al. (MT-Bench) measured a several-point self-preference bias for GPT-4 judging GPT-4 outputs versus blinded humans.
2026 production teams typically cross-judge Claude Opus 4.7 outputs with GPT-5.5 or Gemini 3.1 Pro on subjective dimensions to control for family bias.
LangSmith and Braintrust let you configure ensembles of judge models for pairwise and Likert evaluations.
The 2024 G-Eval and Prometheus 2 papers explicitly recommend rubric-based scoring to reduce open-ended judgment bias.
Cohere's Aya and Mistral's evaluation work both use cross-family judges to publish multi-model benchmark results.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you actually measure self-preference bias in your own eval pipeline?

QWhen does cross-family judging not help much?

QHow does pairwise scoring change the bias profile?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing 'a model can be objective about its own outputs'. The literature is unambiguous: it cannot, and multi-agent makes it worse.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.