HumanEval is the go-to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.

**HumanEval** is a code generation benchmark created by OpenAI. It contains 164 Python programming problems, each consisting of a function signature, a docstring describing what the function should do, and a hidden set of unit tests. The model's job is to generate the function body. Once generated, the code is executed against the unit tests. If all tests pass, the solution is correct. If any test fails, the solution is wrong. There is no partial credit. The primary metric is pass@k, which answers the question: 'If I generate k independent solutions, what is the probability that at least one of them passes all tests?' For example, pass@1 measures how often the model gets it right on the first try, while pass@10 measures how often at least one of ten attempts succeeds. This distinction matters because some models produce correct code unreliably but can find a working solution given enough attempts.

**HumanEval** is a benchmark of 164 Python function-completion problems, each defined by a docstring and function signature. The model generates the function body, and correctness is determined by executing a hidden suite of unit tests against the generated code. The evaluation is entirely execution-based. There is no human judge, no style scoring, and no partial credit. The code either passes every test or it fails. This binary signal is both the strength and the limitation of the benchmark: it catches logical errors perfectly but says nothing about code readability, efficiency, or maintainability. The primary metric is pass@k: the probability that at least one of k independently generated samples passes all tests. An unbiased estimator is used to compute this from n total samples (where n is greater than or equal to k) rather than simply counting successes, which would overestimate performance. pass@1 measures first-try reliability; pass@10 or pass@100 measures whether the model can find a correct solution given multiple attempts. HumanEval+ extends the original by adding more rigorous test cases, catching solutions that passed the original tests through lucky edge-case alignment rather than genuine correctness.