Evaluating a code generation model on real-world tasks beyond HumanEval: which metrics cover correctness, efficiency, and style?
pass@k for correctness via execution, runtime benchmarks for efficiency, LLM-as-judge for style, and SWE-bench Verified for real-world multi-file realism.
Imagine grading a student's programming assignment. You would not just compare their code word for word against the answer key, because there are many ways to write correct code. Instead, you would run their code against test cases to check if it works, time it to see if it is fast enough, and read it to see if it follows good coding practices. That is exactly what pass@k, runtime benchmarks, and a code-review rubric do for evaluating a code generation model.
Detailed answer & concept explanation~3 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the three-axis framing: correctness, efficiency, style. Walk pass@k as execution-based testing that handles code's many correct solutions property. Explain why BLEU and exact match fail. Cover runtime benchmarks for efficiency and LLM-as-judge plus linting for style. Close with SWE-bench Verified as the realism layer that HumanEval misses.
Real products, models, and research that use this idea.
- OpenAI uses HumanEval and SWE-bench as complementary benchmarks: HumanEval for function-level correctness, SWE-bench for real-world engineering capability.
- SWE-bench Verified curates a subset of GitHub issues with verified test patches, filtering out ambiguous or under-specified issues from the original SWE-bench.
- Anthropic evaluates Claude's code generation on internal benchmarks that include multi-file reasoning tasks alongside pass@k on isolated function problems.
- GitHub Copilot measures user acceptance rate in production as a complement to offline pass@k, bridging the gap between benchmark correctness and real-world usefulness.
What an interviewer would ask next. Try answering before peeking at the approach.
Qpass@k gives you a single number. How do you decompose it to find what kinds of problems the model struggles with?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Using BLEU or exact match against reference code, which penalizes correct solutions that use different variable names, control flow, or algorithmic approaches.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.