HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.
A coding benchmark of 164 hand-written Python problems with hidden unit tests. Measures functional correctness by running the generated code rather than string-matching. Pass@1 / pass@10 are reported.
In practice
Default benchmark for coding ability. Interviews probe pass@k semantics and why teams supplement with MBPP and live benchmarks like SWE-bench.
Related topics
Questions that mention this term
- How do you detect…
- What structural property of SWE-bench makes it a harder eval than HumanEval for code agents?
- HumanEval is the go to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.
- Evaluating a code generation model on real world tasks beyond HumanEval: which metrics cover correctness, efficiency, and style?
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
Jailbreak
Adversarial prompt that bypasses an LLM's safety training to elicit normally-refused content.