Why does using the same model family as synthetic-data teacher AND eval judge inflate FT scores? · Zenaique

Attempt it

Quick reference

TL;DR

Same-family teacher and judge create self-preference bias: the judge rewards the style the student inherited, so the score inflates beyond real task quality.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a cooking class where the same chef both writes the recipes and judges the finals. Students who cook exactly in that chef's style get top marks even when the dish is only average, because the judge unconsciously rewards the seasoning and plating they themselves favour. A student who cooks in a different style with equal skill scores lower. The fix is to bring in a different chef to judge, ideally one with a totally different palate, or to compare every dish against a small panel of real diners whose tastes are known. Then you can see how much of the score is real cooking and how much is mimicking the original chef. Without that switch, the school keeps producing graduates who can only please one judge.

Key concepts

LLM-as-judge evaluation Data leakage Synthetic data generation Distillation Held-out evaluation

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: name self-preference bias, explain why style inheritance from the teacher gets rewarded by a same-family judge, prescribe cross-family judging plus a human anchor shard, and use the score gap as a diagnostic.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's published eval methodology recommends cross-family judging and anchors LLM-judge results against human-rated reference sets.
Chatbot Arena pairs models from different families and uses human raters, sidestepping LLM-as-judge bias entirely.
The JudgeLM and Prometheus evaluator papers documented systematic self-preference bias when judges share lineage with the candidate.
Modern flagships like Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro are commonly paired as cross-family judge ensembles on synthetic-SFT eval shards.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you quantify the size of the self-preference bias on your current eval setup?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if you average three judges that are all from the same family?

Sign in or go Pro to see the approach for this follow-up.

QHow do you handle eval when there is no cross-family judge of comparable quality to your teacher?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking a judge from the same family as the teacher 'because it scores reliably'. That reliability is partly self-preference bias dressed as agreement.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.