**MMLU** (Massive Multitask Language Understanding) is a benchmark of about 16,000 multiple-choice questions spread across 57 subjects, from elementary math to professional law and medicine. Each question gives four options and one is correct. A model picks an option and gets graded automatically.
The correct answer is the first option. Its real strength is breadth plus reproducibility. One number summarizes how a model does across a huge range of academic knowledge, and because grading is exact-match on a letter, anyone can re-run it and get the same score.
The weaknesses come in three parts. The multiple-choice format only tests whether a model can recognize the right answer, not whether it can generate a good answer. Because the dataset is public, questions can leak into pre-training data, so a high score may reflect memorization rather than understanding. And a strong MMLU number does not reliably predict how helpful or accurate the model is on open-ended, real tasks.
The other three options are wrong: MMLU is not saturated at 100 percent, it does not refresh monthly, and it does not grade essays. It is a static multiple-choice test.
**MMLU** is a 57-subject, four-option multiple-choice benchmark of roughly 16,000 questions. Option one is correct because it names both the genuine strength and the real weaknesses.
The strength is breadth with reproducibility. It spans humanities, STEM, law, and medicine in one number. Grading is exact-match on a letter, so it is fully automated and anyone can reproduce the score. That made it the default capability headline for years.
The weaknesses are structural. Multiple-choice tests recognition, not generation. A model can pick the right letter without writing a correct, well-reasoned answer. Because the set is public and widely scraped, contamination is real. Questions appear in pre-training corpora, so high scores partly reflect memorization. And recognition accuracy correlates weakly with open-ended quality, the thing production teams actually ship.
The distractors fail on facts. MMLU is not saturated at 100 percent. It does not refresh monthly. It does not grade essays against expert references. Modern practice treats MMLU as one coarse signal, paired with MMLU-Pro, contamination checks, and task-specific evals rather than trusted alone.
Quick reference
TL;DR
MMLU's strength is broad, auto-gradable multiple-choice coverage of 57 subjects. Its weaknesses are recognition-only testing, contamination from public exposure, and weak correlation with open-ended generation quality.
Imagine grading how smart someone is using a giant 57-subject multiple-choice trivia quiz. The good part: it is fast to grade, covers everything from law to biology, and everyone takes the same test, so scores are comparable. The bad part comes in three pieces. First, picking the right bubble shows you can recognize the answer, not that you could write a good essay or hold a real conversation. Second, the quiz is famous and posted online, so a model may have already seen the questions during training and is partly remembering, not reasoning. Third, a high quiz score does not promise the model is actually helpful, honest, or good at the messy open-ended work people really care about.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Evaluating open-ended generation quality such as summaries, chat, or long-form drafting MMLU only scores letter recognition and says nothing about how good a generated answer reads or whether it is faithful.
Ranking frontier models where top scores cluster within a couple of points Saturation plus format noise make tiny gaps meaningless. Move to MMLU-Pro or arena preference signals instead.
Trusting a reported score without any contamination control Public exposure means the number may reflect memorization. Without canary or overlap checks it is not trustworthy.
Evaluating a narrow domain product like a medical or legal copilot A broad 57-subject average hides domain performance. Build a task-specific golden set for the actual workload.
Quick scan · 2–4 min
Everything important, quickly.
5 min: what MMLU measures, its two strengths (breadth, reproducibility), its three weaknesses (recognition only, contamination, weak generation correlation), why the distractors are false, and how MMLU-Pro plus triangulation fix the gaps.
Real-world examples
Real products, models, and research that use this idea.
1Frontier model cards in 2026 (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) report MMLU and MMLU-Pro side by side because plain MMLU lost discrimination at the top.
2MMLU-Pro expanded answers from four to ten options and added reasoning-heavy items specifically to counter saturation and guessing inflation.
3Contamination studies repeatedly find verbatim MMLU questions inside common pre-training corpora like Common Crawl derivatives.
4Chatbot Arena (LMSYS) ELO is now the preferred headline for open-ended quality precisely because MMLU does not measure generation.
5HELM and the Open LLM Leaderboard report MMLU with fixed prompts and shot counts to keep scores reproducible across submissions.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect whether MMLU questions contaminated a model's pre-training data?
Sign in or go Pro to see the approach for this follow-up.
QWhy does MMLU-Pro expand from four options to ten and add reasoning items?
Sign in or go Pro to see the approach for this follow-up.
QIf MMLU measures recognition, what would you pair it with to estimate production quality?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reading a leaderboard MMLU score as a measure of real world capability. It measures recognition over a public, contaminated, multiple-choice set, not open-ended generation quality.