MMLU's strength is broad, auto-gradable multiple-choice coverage of 57 subjects. Its weaknesses are recognition-only testing, contamination from public exposure, and weak correlation with open-ended generation quality.
Imagine grading how smart someone is using a giant 57-subject multiple-choice trivia quiz. The good part: it is fast to grade, covers everything from law to biology, and everyone takes the same test, so scores are comparable. The bad part comes in three pieces. First, picking the right bubble shows you can recognize the answer, not that you could write a good essay or hold a real conversation. Second, the quiz is famous and posted online, so a model may have already seen the questions during training and is partly remembering, not reasoning. Third, a high quiz score does not promise the model is actually helpful, honest, or good at the messy open-ended work people really care about.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MMLU (Massive Multitask Language Understanding) became the default capability headline for large language models because it compresses an enormous range of knowledge into one comparable number. It is a set of roughly 16,000 four-option multiple-choice questions spanning 57 subjects, from elementary mathematics through professional law, medicine, and moral reasoning. A model reads a question, picks a letter, and grading is exact-match. That simplicity is both why it was adopted everywhere and why it misleads when read naively.
This question asks you to identify the option that correctly pairs MMLU's genuine strengths with its genuine weaknesses. The correct answer names breadth and reproducible multiple-choice grading as the strengths, and recognition-only testing, contamination risk, and weak correlation with generation quality as the weaknesses. The deep dive walks each property, why the three distractors are factually wrong, and how the field moved past plain MMLU.
What MMLU actually measures
MMLU is a static bank of multiple-choice questions across 57 domains, grouped into humanities, social sciences, STEM, and a catch-all 'other' category. Each item presents a stem and four labeled options, exactly one correct. The model's job is discrimination: pick the right letter.
Scoring is exact-match on the chosen letter, averaged per subject and then macro-averaged into a single headline accuracy. There is no partial credit, no rubric, no generation. Because the answer space is fixed, grading is deterministic and needs no judge model or human rater.
This design is what makes MMLU a probe of parametric knowledge plus light reasoning. It asks whether the facts and inferences a question needs are recoverable from the model's weights well enough to rank the correct option above three distractors. That is a real and useful signal, but it is a narrow one, and the rest of the deep dive is about that narrowness.
It is also worth being precise about what the headline number is not. It is not a measure of calibration, since the model is never asked how confident it is. It is not a measure of robustness, since each item is presented once in a clean academic format. And it is not a measure of whether the model can apply the recognized fact in a downstream task. Those are separate constructs that demand separate instruments, and conflating them with the MMLU number is the root of most misreadings.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Frontier model cards in 2026 (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) report MMLU and MMLU-Pro side by side because plain MMLU lost discrimination at the top.
- MMLU-Pro expanded answers from four to ten options and added reasoning-heavy items specifically to counter saturation and guessing inflation.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect whether MMLU questions contaminated a model's pre-training data?
Embed canary strings in held-out probes, run n-gram or substring overlap scans against the training corpus, and compare performance on original versus perturbed paraphrases of the same items. A big gap on paraphrases signals memorization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading a leaderboard MMLU score as a measure of real world capability. It measures recognition over a public, contaminated, multiple-choice set, not open-ended generation quality.
60 second bullets to scan on the way to the call.
What MMLU is: 57 subjects, four options, exact-match grading
Why breadth and reproducibility are its core strengths
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.