Identify the judge bias risk when the same model judges its own multi-agent trajectories

MCQ·Hard·4.0 · 0·~1 min·Asked atAi21 Braintrust Uber

Attempt it

**Self-preference bias** is the documented effect that an LLM judge tends to score outputs from its own model family higher than equivalent outputs from a different family. It shows up in single-agent settings (Zheng et al., MT-Bench) and gets sharper in multi-agent because each agent persona is another surface where the judge can recognise its own style. The bias is not about correctness on a math problem (which is checkable). It shows up on the open-ended quality dimensions: reasoning depth, coherence, helpfulness. The judge sees writing that feels familiar (sentence rhythm, paragraph shape, vocabulary choices) and rates it a little higher than blinded human raters would. **Three mitigations that work.** - **Cross-family judging.** Use a judge from a different model family than any agent in the system. Anthropic agents, OpenAI judge (or vice versa). - **Judge ensembling.** Average scores across two or three different judge models so individual family preferences cancel. - **Rubric scoring.** Constrain the judge to specific check items (did the agent call tool X with arg Y? did the critic flag the bug?) rather than open-ended 'how good was this'. Specific checks are harder to bias. The three wrong answers fail for concrete reasons: there is measurable bias (well replicated), JSON failures are not the issue (the bias affects free-text scoring), and latency has nothing to do with self-preference.

**The risk is self-preference bias, amplified by multi-agent style surfaces.** Self-preference is the well-documented effect where an LLM judge gives higher scores to outputs from its own model family than blinded human raters do. The original evidence (Zheng et al. on MT-Bench, follow-up work through 2024-2025) measured the gap at several points on Likert-style quality dimensions. The effect concentrates on subjective dimensions (reasoning depth, helpfulness, coherence) and largely disappears on objective dimensions (final-answer correctness on a math problem) because correctness is checkable independently. **Why multi-agent makes it worse.** Each agent persona has its own writing style, and that style is shaped by the underlying model. A multi-agent trajectory from a single family carries the family's fingerprint across the planner, the executor, and the critic. The judge sees three style surfaces it recognises, not one, and the familiarity bias compounds. A cross-family trajectory exposes more 'foreign' style and gets penalised on the same subjective dimensions. **The three mitigations.** - **Cross-family judge.** If your agents are Claude family, judge with GPT-4o or Gemini. Eliminates the same-family effect at the cost of introducing the new judge's own biases (different shape, but bounded). - **Judge ensemble.** Two or three judges from different families, averaged. Individual biases cancel; remaining noise is acceptable for most production decisions. - **Rubric over Likert.** Replace 'rate this from 1 to 5 on helpfulness' with specific checks: did the planner produce a numbered plan, did the coder call `write_file`, did the critic raise an objection. Specific checks pin the judge to behaviour rather than style. **Why the wrong answers fail.** 'No measurable bias' contradicts replicated empirical evidence. 'JSON failures' confuses output-format errors with judgment errors. 'Runs slower' is unrelated to bias and not even true.

TL;DR

Self-preference bias inflates judge scores when the judge shares a family with the agents; multi-agent amplifies it because each persona is another style surface.

Memory aid

Easy to grasp

Imagine a baking contest where the judge happens to be the head baker at one of the competing bakeries. Even if the judge tries to be fair, they will recognise their own style of icing, their own kind of crumb, their own preferred sweetness, and tend to score familiar cakes a little higher. The cakes are not better; they just feel right. Now imagine the contest has three courses per entry, so the judge sees their own style three times per competitor. The familiarity advantage compounds. That is what happens when an LLM judges multi-agent trajectories from its own model family.

Key concepts

Multi-Agent Systems

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

LLM-as-judge is now the default for evaluating open-ended agent outputs because human evaluation does not scale. The honest catch is that the judge is itself an LLM, and LLMs have measurable, replicated biases when evaluating their own family's outputs. In single-agent settings this is annoying. In multi-agent settings it is amplified, because each agent persona presents another style surface for the bias to attach to.

This question is testing whether you know the bias exists, why it gets worse for multi-agent, and what to do about it. The wrong answers are each a different category of misunderstanding: denial of evidence, format confusion, and a distractor about runtime.

One-line summary: the bias is real, it concentrates on subjective dimensions, multi-agent makes it worse, and the mitigation stack is cross-family judges, ensembling, and rubric scoring.

What self-preference bias is, in measurable terms

The original measurement

Zheng et al. (NeurIPS 2023, MT-Bench) showed that GPT-4, when used as a judge to score GPT-4 outputs against human-annotated preferences, systematically rated its own family higher than blinded human raters did. The effect was several Likert points on subjective dimensions like helpfulness and reasoning depth. Follow-up work through 2024-2025 replicated the pattern across Claude, Gemini, and open-weight families.

Where it concentrates

The bias is largest on dimensions that are not externally verifiable: helpfulness, coherence, depth of reasoning. It shrinks toward zero on dimensions that are checkable: final-answer correctness on math problems, code that compiles and passes tests, exact-match QA. The mechanism is consistent with stylistic familiarity: the judge's training distribution gives it a prior over what 'good output' looks like, and that prior matches same-family outputs more closely.

What it is not

It is not 'a model thinks it is great and rates everything 5/5'. The shape is comparative: same-family scores are inflated relative to a blinded baseline. The judge can still rank correctly within a family; it just rates a same-family candidate higher than an equivalent cross-family candidate.

Why multi-agent amplifies the effect

The mitigation stack, ordered by cost-effectiveness

When the mitigations do not help, and the senior take

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Zheng et al. (MT-Bench) measured a several-point self-preference bias for GPT-4 judging GPT-4 outputs versus blinded humans.
2026 production teams typically cross-judge Claude Opus 4.7 outputs with GPT-5.5 or Gemini 3.1 Pro on subjective dimensions to control for family bias.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you actually measure self-preference bias in your own eval pipeline?

Run the same set of trajectories through judges from different model families, compare scores per trajectory, and fit a per-judge bias term against the consensus or a blinded human gold set. The judge whose scores systematically rise for same-family trajectories is showing the bias by definition.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing 'a model can be objective about its own outputs'. The literature is unambiguous: it cannot, and multi-agent makes it worse.

60 second bullets to scan on the way to the call.

What self-preference bias is and where it has been measured
Why the bias is largest on subjective dimensions and smallest on externally checkable ones

Primary sources. Browse if you want the original framing.