pass@k for correctness via execution, runtime benchmarks for efficiency, LLM-as-judge for style, and SWE-bench Verified for real-world multi-file realism.
Imagine grading a student's programming assignment. You would not just compare their code word for word against the answer key, because there are many ways to write correct code. Instead, you would run their code against test cases to check if it works, time it to see if it is fast enough, and read it to see if it follows good coding practices. That is exactly what pass@k, runtime benchmarks, and a code-review rubric do for evaluating a code generation model.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Code evaluation is fundamentally different from natural language evaluation because code has a unique property: there are many correct implementations of the same specification. A function that sorts a list can use quicksort, mergesort, timsort, or a built-in, with different variable names and control flow in each case. Any evaluation method that compares generated code textually against a reference will penalize correct alternatives.
This property drives the entire design of a code evaluation stack: correctness must be execution-based (pass@k), efficiency must be measured by runtime or complexity analysis, and style must be judged by a rubric rather than textual similarity. SWE-bench adds the realism dimension that isolated function benchmarks miss.
Correctness: pass@k and execution-based testing
pass@k generates k candidate solutions for each problem, runs each against a test suite, and computes the probability that at least one passes all tests. The metric acknowledges that models are stochastic: a model that generates 10 solutions where 3 are correct is genuinely useful, even though 7 are wrong.
The unbiased estimator for pass@k avoids overestimation from naive sampling. For large k, the naive approach of generating k samples and checking any-pass gives biased results. The analytical formula computes the exact probability from the number of correct samples observed.
Test suite quality is the bottleneck. If the test suite has only 2 test cases and both are happy-path, pass@k will be inflated because solutions that fail on edge cases still pass. Strong test suites with edge cases, boundary conditions, and adversarial inputs produce pass@k scores that correlate with real-world correctness.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI uses HumanEval and SWE-bench as complementary benchmarks: HumanEval for function-level correctness, SWE-bench for real-world engineering capability.
- SWE-bench Verified curates a subset of GitHub issues with verified test patches, filtering out ambiguous or under-specified issues from the original SWE-bench.
What an interviewer would ask next. Try answering before peeking at the approach.
Qpass@k gives you a single number. How do you decompose it to find what kinds of problems the model struggles with?
Slice by problem category (string manipulation, algorithms, data structures, API usage). Compute pass@k per slice. If the model passes 90% of string problems but 30% of graph algorithms, you know where to focus improvement. Also slice by test complexity: problems with 2 tests versus 20 tests reveal whether the model handles edge cases.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using BLEU or exact match against reference code, which penalizes correct solutions that use different variable names, control flow, or algorithmic approaches.
60 second bullets to scan on the way to the call.
Why execution-based testing is necessary for code correctness
How pass@k works and why it handles the many correct solutions property of code
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.