LLM As Judge Biases

Spot the flaw in this LLM evaluation setup that uses GPT-4 as judge

What is self-preference bias in LLM-as-judge setups?

Predict which answer a vanilla GPT-4 judge prefers: short+correct vs long+off-topic

What is the standard mitigation for position bias in pairwise LLM-judge evaluation?

What is position bias in pairwise LLM-judge evaluation?

How do you detect and mitigate length bias when using an LLM judge in a production eval pipeline?

Why do LLM judges tend to rate longer answers higher, even when verbosity reduces quality?

Which statements correctly describe the benefits and limitations of a panel of judges eval setup?

How does a panel of judges setup reduce LLM-as-judge bias?