What are the strengths and weaknesses of MMLU as an LLM benchmark?
MMLU's strength is broad, auto-gradable multiple-choice coverage of 57 subjects. Its weaknesses are recognition-only testing, contamination from public exposure, and weak correlation with open-ended generation quality.
Imagine grading how smart someone is using a giant 57-subject multiple-choice trivia quiz. The good part: it is fast to grade, covers everything from law to biology, and everyone takes the same test, so scores are comparable. The bad part comes in three pieces. First, picking the right bubble shows you can recognize the answer, not that you could write a good essay or hold a real conversation. Second, the quiz is famous and posted online, so a model may have already seen the questions during training and is partly remembering, not reasoning. Third, a high quiz score does not promise the model is actually helpful, honest, or good at the messy open-ended work people really care about.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: what MMLU measures, its two strengths (breadth, reproducibility), its three weaknesses (recognition only, contamination, weak generation correlation), why the distractors are false, and how MMLU-Pro plus triangulation fix the gaps.
Real products, models, and research that use this idea.
- Frontier model cards in 2026 (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) report MMLU and MMLU-Pro side by side because plain MMLU lost discrimination at the top.
- MMLU-Pro expanded answers from four to ten options and added reasoning-heavy items specifically to counter saturation and guessing inflation.
- Contamination studies repeatedly find verbatim MMLU questions inside common pre-training corpora like Common Crawl derivatives.
- Chatbot Arena (LMSYS) ELO is now the preferred headline for open-ended quality precisely because MMLU does not measure generation.
- HELM and the Open LLM Leaderboard report MMLU with fixed prompts and shot counts to keep scores reproducible across submissions.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect whether MMLU questions contaminated a model's pre-training data?
QWhy does MMLU-Pro expand from four options to ten and add reasoning items?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Reading a leaderboard MMLU score as a measure of real world capability. It measures recognition over a public, contaminated, multiple-choice set, not open-ended generation quality.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.