pass@k is the chance at least one of k code samples passes the unit tests. The unbiased estimator generates n more than k samples, counts c passes, and computes the probability in closed form for low variance.
Imagine a student who gets several tries to solve a puzzle, and they pass if any single attempt works. pass@k asks: with k tries, what is the chance at least one succeeds? You could just hand them k attempts and see, but that is noisy. A lucky or unlucky batch swings the number a lot. So instead you let them make many attempts and count how many worked. Then you work out how likely k random tries would include a winner. Using every attempt you collected gives a far steadier number than judging from one small handful. Code-eval teams use the same idea to grade a model on programming problems.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
pass@k is the metric that made code generation benchmarks meaningful. Earlier code metrics borrowed BLEU and exact match from machine translation. But a program that differs character by character from a reference can still be perfectly correct, and a program one token away can be broken. Surface overlap with a reference solution is the wrong target. The only reliable test of a program is whether it runs and produces the right behavior on the inputs you care about.
pass@k follows that idea. It is execution based: you run each candidate against unit tests and call it correct only if every test passes. And it is best of k: it rewards a model for having at least one working solution among k tries. That is how developers and coding agents use these tools in practice. You generate, filter by tests, and keep a winner.
What pass@k measures
pass@k is the probability that, given k independent samples from the model for a single problem, at least one of them passes all of the problem's unit tests. You compute this quantity per problem and then average across every problem in the benchmark to get the single reported number. The unit of correctness is the whole test suite for a problem, not individual assertions: a candidate that fails one test fails the problem.
The metric is deliberately a best of k score, not an average over k. Code generation is forgiving in practice. A developer or a coding agent can generate several candidates, run the test suite locally, and keep whichever one works. The economic value of a model is therefore the chance that some sample in the budget is correct, not the chance that a random single sample is correct. An average would punish a model that is usually wrong but reliably produces one correct solution. Best of k correctly rewards it.
Common reporting uses k in {1, 10, 100}. pass@1 is the single-shot accuracy and is the hardest setting, because you get exactly one attempt. pass@100 reflects how good the model is when you can afford many tries and filter by tests. The gap between pass@1 and pass@100 is itself diagnostic. A wide gap means the model has the right idea somewhere in its distribution but cannot rank it first. That points at reranking, verification, or decoding fixes rather than more pretraining.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import numpy as np
def pass_at_k(n: int, c: int, k: int) -> float:
# n = samples generated, c = samples that passed, k = budget
if n - c < k:
return 1.0
# 1 - product form of C(n-c, k) / C(n, k), numerically stable
return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))Real products, models, and research that use this idea.
- HumanEval, the OpenAI Codex benchmark, defines pass@k with the unbiased estimator and remains a default code-eval metric.
- MBPP and HumanEval+ also use pass@k. The EvalPlus leaderboard ranks models by pass@1 with greedy decoding.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the closed-form estimator lower variance than computing pass@k from a single draw of k samples?
It uses all n generations as evidence rather than discarding the rest. The estimator marginalizes over every possible k-subset analytically, so it is the exact expectation rather than one Monte Carlo sample of it.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Generating exactly k samples and reporting the raw hit rate. That estimate has high variance. Generate n more than k, count passes, and use the closed-form estimator instead.
60 second bullets to scan on the way to the call.
What pass@k measures and why best of k matches real code usage
Why the naive exactly k estimate has high variance
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.