Evaluating code generation needs three different kinds of metrics because code has three different qualities that matter: does it work, is it fast, and is it readable?
**Correctness: pass@k.** Run the generated code against a test suite. pass@k measures the probability that at least one of k generated samples passes all tests. This is execution-based evaluation, which is the only reliable way to check correctness. Textual comparison (BLEU, exact match) fails because many different code implementations are equally correct.
**Efficiency: runtime benchmarks.** Time the generated solutions on standardized inputs. Compare against a baseline implementation. For algorithm-focused tasks, verify the Big-O complexity matches the expected bound.
**Style: linting plus LLM-as-judge.** Linting catches mechanical issues (unused variables, formatting violations). LLM-as-judge with a code-review rubric catches readability, naming conventions, and adherence to project patterns.
**Realism: SWE-bench Verified.** HumanEval only tests isolated function completion. SWE-bench Verified uses real GitHub issues that require reading existing code across multiple files. This is where you test whether the model can work on a real codebase, not just write a function from scratch.
**Correctness** requires execution-based testing. pass@k generates k candidate solutions, runs each against a test suite, and computes the probability that at least one passes. This handles the fundamental property of code: many implementations are correct. BLEU and exact match penalize valid alternatives.
**Efficiency** needs runtime benchmarks on standardized inputs, or Big-O analysis of the generated algorithm. A correct but O(n^3) solution is a failure on a task that expects O(n log n).
**Style** combines automated linting (formatting, unused imports, naming conventions) with LLM-as-judge using a code-review rubric (readability, idiomatic patterns, project convention adherence). Linting catches mechanical violations; the judge catches higher-level quality.
**Realism** is the missing dimension in most code evals. HumanEval tests isolated function completion. SWE-bench Verified requires multi-file reasoning against real GitHub issues: reading existing code, understanding the bug or feature request, and producing a patch that integrates with the codebase. This level of realism is what separates a toy eval from a production-relevant one.
Quick reference
TL;DR
pass@k for correctness via execution, runtime benchmarks for efficiency, LLM-as-judge for style, and SWE-bench Verified for real-world multi-file realism.
Imagine grading a student's programming assignment. You would not just compare their code word for word against the answer key, because there are many ways to write correct code. Instead, you would run their code against test cases to check if it works, time it to see if it is fast enough, and read it to see if it follows good coding practices. That is exactly what pass@k, runtime benchmarks, and a code-review rubric do for evaluating a code generation model.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Evaluating code explanation or documentation generation, not code generation. pass@k requires executable code. For explanation quality, use LLM-as-judge with a clarity and accuracy rubric instead.
A rapid prototyping phase where you need a quick signal on model quality. SWE-bench Verified is expensive to run. HumanEval pass@1 gives a fast directional signal that is sufficient for early model selection.
Evaluating code completion suggestions in an IDE context. pass@k measures whole-function generation. IDE completion requires measuring acceptance rate, time to accept, and edit distance from the suggestion, which are interaction-level metrics.
Quick scan · 2–4 min
Everything important, quickly.
Open with the three-axis framing: correctness, efficiency, style. Walk pass@k as execution-based testing that handles code's many correct solutions property. Explain why BLEU and exact match fail. Cover runtime benchmarks for efficiency and LLM-as-judge plus linting for style. Close with SWE-bench Verified as the realism layer that HumanEval misses.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI uses HumanEval and SWE-bench as complementary benchmarks: HumanEval for function-level correctness, SWE-bench for real-world engineering capability.
2SWE-bench Verified curates a subset of GitHub issues with verified test patches, filtering out ambiguous or under-specified issues from the original SWE-bench.
3Anthropic evaluates Claude's code generation on internal benchmarks that include multi-file reasoning tasks alongside pass@k on isolated function problems.
4GitHub Copilot measures user acceptance rate in production as a complement to offline pass@k, bridging the gap between benchmark correctness and real-world usefulness.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
Qpass@k gives you a single number. How do you decompose it to find what kinds of problems the model struggles with?
Sign in or go Pro to see the approach for this follow-up.
QSWE-bench Verified uses real GitHub issues. What bias does this introduce?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Using BLEU or exact match against reference code, which penalizes correct solutions that use different variable names, control flow, or algorithmic approaches.