MMLU appears on every model leaderboard. Describe what it tests and its biggest blind spot. · Zenaique

Attempt it

MMLU (Massive Multitask Language Understanding) tests knowledge across 57 academic subjects using four-choice multiple-choice questions. It measures how well a model can recognize the correct answer from a fixed set of options. Its biggest blind spot: it only tests recognition, not generation. A model can score 90% on MMLU and still produce poor free-form responses because selecting the right option from four choices is a fundamentally different skill from generating a coherent, helpful paragraph from scratch.

Your answer

Quick reference

TL;DR

MMLU tests knowledge breadth across 57 subjects via four-choice questions, but its blind spot is that recognition does not prove generation ability.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a driving test that only has written multiple-choice questions and never asks you to actually drive a car. You could memorize all the road rules and score 100% without ever touching a steering wheel. MMLU is like that written test for AI models. It checks whether the model can pick the right answer from four options across many subjects. But it never checks whether the model can actually write a helpful explanation, handle a follow-up question, or admit when it does not know something. That is the blind spot.

Key concepts

MMLU Multiple-Choice Evaluation Knowledge Breadth Benchmark Contamination

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define MMLU as 57-subject four-choice benchmark, explain the recognition versus generation blind spot with a concrete example, discuss contamination vulnerability, mention MMLU-Pro as a partial fix, and frame MMLU as one dimension of a multi-dimensional evaluation strategy.

Real-world examples

Real products, models, and research that use this idea.

Every major model release in 2026 includes an MMLU or MMLU-Pro score in its announcement, making it the most recognized benchmark number in the industry.
MMLU-Pro was introduced as a harder successor with ten-choice questions and chain-of-thought requirements, addressing some of MMLU's contamination and difficulty ceiling issues.
Teams evaluating models for enterprise deployment often pair MMLU scores with Arena-style or MT-Bench evaluations to capture both knowledge breadth and generation quality.
Open-weight model leaderboards on Hugging Face prominently feature MMLU scores, making it a key comparison metric for the open-source community.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does MMLU-Pro improve on MMLU, and does it fully solve the recognition-generation gap?

Sign in or go Pro to see the approach for this follow-up.

QHow would you detect whether a model's high MMLU score is inflated by benchmark contamination?

Sign in or go Pro to see the approach for this follow-up.

QIf MMLU scores are saturating for frontier models, what comes next?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Equating a high MMLU score with overall model quality, when the benchmark only measures recognition of correct answers, not open-ended generation.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.