MMLU appears on every model leaderboard. Describe what it tests and its biggest blind spot.
MMLU tests knowledge breadth across 57 subjects via four-choice questions, but its blind spot is that recognition does not prove generation ability.
Imagine a driving test that only has written multiple-choice questions and never asks you to actually drive a car. You could memorize all the road rules and score 100% without ever touching a steering wheel. MMLU is like that written test for AI models. It checks whether the model can pick the right answer from four options across many subjects. But it never checks whether the model can actually write a helpful explanation, handle a follow-up question, or admit when it does not know something. That is the blind spot.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
If you have read any model comparison in the last three years, you have seen an MMLU score. It has become the default number that journalists, researchers, and hiring managers cite when they want a quick measure of how 'smart' a model is.
This deep dive explains what MMLU actually tests, why it became so dominant, where it fails, and what the evaluation community is building to supplement it.
What MMLU contains and how it is scored
MMLU is a benchmark of 14,042 multiple-choice questions spanning 57 academic subjects. The subjects range across STEM (abstract algebra, college mathematics, computer science), humanities (philosophy, world religions, US history), social sciences (economics, psychology, sociology), and professional domains (clinical knowledge, medical genetics, professional law).
Each question presents four options labeled A through D. The model's job is to assign the highest probability to the correct option. Scoring is straightforward: the percentage of questions answered correctly, either overall or broken down by subject.
The benchmark includes few-shot examples (typically five) that are prepended to each question to give the model context about the expected format. This few-shot setup means the model does not need to be instruction-tuned to take the test; even a base model can be evaluated by conditioning on the examples.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Every major model release in 2026 includes an MMLU or MMLU-Pro score in its announcement, making it the most recognized benchmark number in the industry.
- MMLU-Pro was introduced as a harder successor with ten-choice questions and chain-of-thought requirements, addressing some of MMLU's contamination and difficulty ceiling issues.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does MMLU-Pro improve on MMLU, and does it fully solve the recognition-generation gap?
MMLU-Pro uses ten options instead of four and requires chain-of-thought reasoning, which raises the difficulty ceiling and reduces guessing. But it is still multiple-choice, so the fundamental recognition versus generation gap remains.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Equating a high MMLU score with overall model quality, when the benchmark only measures recognition of correct answers, not open-ended generation.
60 second bullets to scan on the way to the call.
State that MMLU stands for Massive Multitask Language Understanding
Name the format as four-choice multiple-choice across 57 academic subjects
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.