LLM As Judge

Which statement best describes what each of the three LLM evaluation modes covers?

Spot the flaw in this LLM evaluation setup that uses GPT-4 as judge

What is self-preference bias in LLM-as-judge setups?

Spot the design flaws in this LLM evaluation rubric

Which properties characterize a well-designed LLM eval rubric?

Explain faithfulness vs answer relevance in RAG evaluation with concrete examples

Predict which answer a vanilla GPT-4 judge prefers: short+correct vs long+off-topic

What is the standard mitigation for position bias in pairwise LLM-judge evaluation?

What is position bias in pairwise LLM-judge evaluation?

How do you detect and mitigate length bias when using an LLM judge in a production eval pipeline?

Why do LLM judges tend to rate longer answers higher, even when verbosity reduces quality?

How do you navigate the latency, cost, quality triangle when choosing an eval approach?

Order the three eval approaches from cheapest/fastest to most expensive/slowest

What is the primary reason for running LLM judges at temperature=0 in an eval pipeline?

Which statements correctly describe the benefits and limitations of a panel of judges eval setup?

How does a panel of judges setup reduce LLM-as-judge bias?

Tradeoffs between pairwise LLM-judge comparisons and scalar scoring rubrics

What is the primary scalability limitation of pairwise LLM-judge evaluation compared to scalar scoring?

What is LLM-as-judge and why did it emerge as an evaluation approach?

What does Cohen's kappa measure in human LLM evaluation?

Explain G-Eval's mechanism and why it improves over direct-score LLM judges

What is G-Eval and how does it differ from a direct-score LLM judge?

Design an LLM judge prompt with anchor examples and explain how they reduce score variance

Why do anchor examples in an LLM judge prompt dramatically reduce score variance?

How do you detect quality drift in a production LLM system without ground-truth labels?