MMLU (Massive Multitask Language Understanding)
Also known as: Massive Multitask Language Understanding
Multiple-choice benchmark across 57 academic subjects; the standard "raw knowledge" headline number.
A benchmark of ~16k multiple-choice questions across 57 subjects (math, history, law, medicine, etc.) used to measure broad academic knowledge of LLMs. Standard headline metric for model releases.
In practice
Most-cited benchmark; also most-gamed. Senior interviews probe its weakness (contamination, MCQ artifacts) and why MMLU-Pro emerged.
How it compares
MMLU tests academic knowledge via multiple-choice; HumanEval tests code generation with executable tests.
Related topics
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.