MMLU (Massive Multitask Language Understanding)
Also known as: Massive Multitask Language Understanding
Multiple-choice benchmark across 57 academic subjects; the standard "raw knowledge" headline number.
A benchmark of ~16k multiple-choice questions across 57 subjects (math, history, law, medicine, etc.) used to measure broad academic knowledge of LLMs. Standard headline metric for model releases.
In practice
Most-cited benchmark; also most-gamed. Senior interviews probe its weakness (contamination, MCQ artifacts) and why MMLU-Pro emerged.
How it compares
MMLU tests academic knowledge via multiple-choice; HumanEval tests code generation with executable tests.
Related topics
Related terms
Hallucination
When a model confidently makes up something that isn't true.
LLM Evaluation
Measuring whether an LLM does what you want, beyond "looks fine to me".
Perplexity
Exp(average cross-entropy) on held-out text; lower means the model is less surprised by real data.
Reasoning Model
An LLM trained to reason at length internally before answering. Slower and more expensive, but much better on hard problems.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.