Benchmarks

Measuring whether a vision-language model is actually good: what goes on the scorecard

Match each multimodal benchmark to the capability it actually measures

A model aces a benchmark with suspiciously high scores. The first hypothesis is contamination. Explain what that means to a non-technical PM.

Every model release cites benchmark numbers. Define what a benchmark actually is and name two things it cannot tell you.

TruthfulQA is designed to catch a specific model failure. Describe what it tests and why larger models sometimes do worse.

MT-Bench is used alongside Chatbot Arena. Describe what MT-Bench tests that single turn benchmarks miss.

MMLU appears on every model leaderboard. Describe what it tests and its biggest blind spot.

HumanEval is the go to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.

Chatbot Arena ranks models by Elo rating. Explain the Elo system to someone who has never played competitive chess.

Why does Chatbot Arena (LMSYS) give a different picture of model quality than static benchmarks like MMLU?

AlpacaEval 2 reports a 'win rate' for each model. Against what baseline and how is the winner decided?

Walk through SWE-bench, AgentBench, GAIA, and TAU-bench: what they measure, their shared blind spots, and how to interpret agent benchmark numbers.

What structural property of SWE-bench makes it a harder eval than HumanEval for code agents?

What makes multi-turn dialog evaluation harder than single turn, and how does MT-Bench address it?

Explain benchmark contamination and its effect on reported LLM capability scores

Which methods are practical approaches for detecting train eval data leakage in LLM benchmarks?

How does Chatbot Arena compute ELO rankings, and what bias affects its validity?

What is the primary validity threat to Chatbot Arena ELO rankings as a measure of general model quality?

Analyze MMLU's strengths and failure modes as an LLM benchmark

What are the strengths and weaknesses of MMLU as an LLM benchmark?