MMLU's strength is broad, auto-gradable multiple-choice coverage of 57 subjects. Its weaknesses are recognition-only testing, contamination from public exposure, and weak correlation with open-ended generation quality.
Imagine grading how smart someone is using a giant 57-subject multiple-choice trivia quiz. The good part: it is fast to grade, covers everything from law to biology, and everyone takes the same test, so scores are comparable. The bad part comes in three pieces. First, picking the right bubble shows you can recognize the answer, not that you could write a good essay or hold a real conversation. Second, the quiz is famous and posted online. So a model may have already seen the questions during training and is partly remembering, not reasoning. Third, a high quiz score does not promise the model is helpful, honest, or good at the messy open-ended work people care about.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MMLU (Massive Multitask Language Understanding) became the default capability headline for large language models because it compresses an enormous range of knowledge into one comparable number. It is a set of roughly 16,000 four-option multiple-choice questions spanning 57 subjects, from elementary mathematics through professional law, medicine, and moral reasoning. A model reads a question, picks a letter, and grading is exact-match. That simplicity is both why it was adopted everywhere and why it misleads when read naively.
Option A names breadth and reproducible multiple-choice grading as the strengths. It names recognition-only testing, contamination risk, and weak correlation with generation quality as the weaknesses.
What MMLU measures
MMLU is a static bank of multiple-choice questions across 57 domains, grouped into humanities, social sciences, STEM, and a catch-all 'other' category. Each item presents a stem and four labeled options, exactly one correct. The model's job is discrimination: pick the right letter.
Scoring is exact-match on the chosen letter, averaged per subject and then macro-averaged into a single headline accuracy. There is no partial credit, no rubric, no generation. Because the answer space is fixed, grading is deterministic and needs no judge model or human rater.
This design is what makes MMLU a probe of parametric knowledge plus light reasoning. It asks whether the facts and inferences a question needs are recoverable from the model's weights well enough to rank the correct option above three distractors. That is a real and useful signal, but it is a narrow one.
The headline number also leaves out three things. It is not a measure of calibration, since the model is never asked how confident it is. It is not a measure of robustness, since each item is presented once in a clean academic format. And it is not a measure of whether the model can apply the recognized fact in a downstream task. Those are separate constructs that need separate instruments. Conflating them with the MMLU number causes most misreadings.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Older frontier model cards reported MMLU and MMLU-Pro side by side because plain MMLU lost discrimination at the top.
- MMLU-Pro expanded answers from four to ten options and added reasoning-heavy items specifically to counter saturation and guessing inflation.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect whether MMLU questions contaminated a model's pre-training data?
Embed canary strings in held-out probes, run n-gram or substring overlap scans against the training corpus, and compare performance on original versus perturbed paraphrases of the same items. A big gap on paraphrases signals memorization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading a leaderboard MMLU score as a measure of real world capability. It measures recognition over a public, contaminated, multiple-choice set, not open-ended generation quality.
60 second bullets to scan on the way to the call.
What MMLU is: 57 subjects, four options, exact-match grading
Why breadth and reproducibility are its core strengths
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.