HumanEval is the go to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.
HumanEval gives the model 164 Python function-completion tasks and checks correctness by running unit tests, reporting pass@k as the primary metric.
Imagine a cooking competition where the judge does not taste the food or care how it looks. Instead, the judge runs the dish through a machine that checks 'Does it have the right temperature? The right ingredients? The right texture?' If every check passes, the dish is correct. If any check fails, the dish is wrong. HumanEval works the same way with code. The model writes a Python function, and a set of automated tests checks whether the function produces the right outputs for every given input. No partial credit, no style points.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
If MMLU is the default knowledge benchmark, HumanEval is the default code generation benchmark. Every model that claims coding ability publishes a HumanEval score. Understanding what the benchmark actually measures, and what it misses, is essential for interpreting those claims.
This deep dive covers the benchmark design, the pass@k metric, known limitations, and the extensions that address those limitations.
The benchmark design
HumanEval was introduced alongside the Codex paper by OpenAI in 2021. It contains 164 hand-written Python programming problems, each consisting of a function signature, a docstring describing the expected behavior, and a hidden suite of unit tests.
The model receives the signature and docstring and must generate the function body. The generated code is then executed in a sandboxed environment against the unit tests. If all tests pass, the solution is marked correct. If any test fails, the solution is marked incorrect. There is no partial credit, no human review, and no style assessment.
The problems range in difficulty from simple string manipulation to moderate algorithmic challenges. They are closer to LeetCode easy/medium problems than to real-world software engineering tasks. Each problem is self-contained: no imports, no multi-file context, no database connections.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI introduced HumanEval alongside the Codex paper in 2021, and it has remained the most cited code generation benchmark through 2026.
- HumanEval+ extends the original with stricter test suites, catching edge-case failures that slipped through the original tests.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the unbiased estimator for pass@k preferred over simply computing the success rate raised to the power of k?
The naive estimator (success_rate^k) overestimates performance because it assumes independence between samples. The combinatorial estimator accounts for the actual number of correct and incorrect samples without the independence assumption.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming HumanEval measures code quality or style when it only checks functional correctness through unit test execution.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.