Analyze MMLU's strengths and failure modes as an LLM benchmark

**MMLU** (Massive Multitask Language Understanding) is a benchmark of multiple-choice questions across 57 subjects, from elementary math to US history to professional medicine. A model reads a question and four options and picks one. You score it by exact match, so grading is automatic and reproducible. Its strengths are real. It is broad, covering many domains in one number. It is cheap and fully reproducible because there is no human grading. And everyone reports it, so you can compare models on a common scale. The weaknesses are why you should not trust it for shipping decisions. First, it tests recognition: picking the right option is easier than writing a correct answer from scratch. Second, the questions are public and old, so they leak into training data and models memorize them instead of reasoning. Third, top models now all score in the high 80s, so the benchmark no longer separates them. Most importantly, MMLU does not look like your task. Your users summarize documents, write code, or ask support questions. A high MMLU score does not predict whether the model is good at any of those. The fix is to build a small task-specific golden set: real examples from your domain, graded against what good output looks like for you.

**MMLU** measures multiple-choice knowledge recognition across 57 academic and professional domains. Its strengths are breadth, exact reproducible scoring, and universal reporting for cross-model comparison. But it is a weak signal for production readiness, for four reasons. Contamination: the questions are public and predate most model training cutoffs. They leak into pre-training data, so scores partly measure memorization, not capability. Saturation: frontier models now cluster in the high 80s near the human ceiling. The benchmark has lost discriminative power at the top. Distribution mismatch is the deepest problem. MMLU is short academic MCQs. Your production task is open-ended generation: summarization, multi-turn support, tool-using agents, RAG over your own documents. The recognition versus generation gap means a model can pick the right option without producing a correct, grounded, well-formatted answer. Prompt-format sensitivity compounds this: the same model scores several points differently depending on option ordering, answer-letter formatting, and few-shot template, so a single reported number hides real variance. What to build instead: a task-specific golden set. Sample real queries from your own domain and define rubric dimensions that match what your users need, such as faithfulness, format, and tone. Grade with exact match where ground truth exists, and a calibrated judge where it does not. That eval predicts your product. MMLU does not.