HumanEval is the go-to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct. · Zenaique

Attempt it

HumanEval presents 164 Python function-completion problems. Each problem gives the model a docstring and a function signature, and the model generates the function body. Correctness is determined by executing a suite of unit tests against the generated code. The primary metric is pass@k: the probability that at least one of k generated samples passes all tests. This execution-based evaluation is what makes HumanEval distinctive. There is no partial credit and no style judgment. The code either passes the tests or it does not.

Your answer

Quick reference

TL;DR

HumanEval gives the model 164 Python function-completion tasks and checks correctness by running unit tests, reporting pass@k as the primary metric.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a cooking competition where the judge does not taste the food or care how it looks. Instead, the judge runs the dish through a machine that checks 'Does it have the right temperature? The right ingredients? The right texture?' If every check passes, the dish is correct. If any check fails, the dish is wrong. HumanEval works the same way with code. The model writes a Python function, and a set of automated tests checks whether the function produces the right outputs for every given input. No partial credit, no style points.

Key concepts

HumanEval pass@k Unit Test Execution Code Generation

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: describe the 164-problem function-completion setup, explain execution-based evaluation with unit tests, define pass@k with the unbiased estimator, discuss limitations (small set, simple problems, no style scoring), and name HumanEval+ and SWE-bench as extensions.

Real-world examples

Real products, models, and research that use this idea.

OpenAI introduced HumanEval alongside the Codex paper in 2021, and it has remained the most cited code generation benchmark through 2026.
HumanEval+ extends the original with stricter test suites, catching edge-case failures that slipped through the original tests.
SWE-bench, introduced as a complement to HumanEval, tests repository-level changes on real GitHub issues, addressing the gap between function-level and project-level coding ability.
Model release announcements for code-specialized models routinely report HumanEval pass@1 scores as a headline metric alongside SWE-bench performance.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the unbiased estimator for pass@k preferred over simply computing the success rate raised to the power of k?

Sign in or go Pro to see the approach for this follow-up.

QHow does SWE-bench complement HumanEval, and what new challenges does it introduce?

Sign in or go Pro to see the approach for this follow-up.

QWhat does it mean when a model has high pass@100 but low pass@1?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming HumanEval measures code quality or style when it only checks functional correctness through unit test execution.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.