Explain the pass@k metric for code generation evaluation. Why is the naive estimator (randomly pick k samples and check if any pass) high variance, and how does the unbiased estimator correct for this?
pass@k is the chance at least one of k code samples passes the tests. Estimate it from n samples with the unbiased Codex formula, not a single random draw of k, which is high-variance.
Imagine a student who solves a tricky problem correctly about a third of the time. pass@k asks: if they get k attempts and we keep the best one, how often does at least one attempt work? You could measure this the lazy way: have them try exactly k times and see if any passed. But that's a coin flip with few tosses, so the number bounces around wildly between problems. The smarter way is to let them try many more times, count how many attempts succeeded overall, and do the arithmetic for what k tries would give. Using all those extra attempts makes the estimate steady instead of jumpy. It also reminds you that one lucky success out of many is not the same as a student who reliably gets it right.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
pass@k is the metric that made functional code evaluation rigorous. Before it, code models were scored with text-overlap metrics like BLEU, which reward looking like a reference solution rather than actually working. Two programs can be byte for byte different yet both correct, and a one-character diff can flip a correct program to a broken one. Text overlap cannot see that. pass@k threw it out: a generation counts only if it passes the full unit-test suite when executed.
This question is hard because the metric is easy to state and easy to estimate wrong. The definition is one sentence. The trap is in the estimator. A senior engineer must explain why the obvious plug-in formula is biased, why drawing k samples once is high-variance, and why the closed-form estimator over n samples is the correct tool. Getting this wrong is not a rounding error. It can reorder a leaderboard.
The deep dive walks the definition, the two failure modes of naive estimation, the derivation of the unbiased estimator, its numerically stable implementation, and the reliability story that a single pass@k number hides.
What pass@k actually measures
For a single problem, pass@k is the probability that at least one of k independently sampled generations passes every unit test. You compute it per problem and average across the benchmark. A generation passes only if the executed code clears the entire test suite. There is no partial credit, and there is no notion of being close. This is the defining feature of functional evaluation: correctness is decided by execution, not by surface similarity to a reference.
The choice of k is the whole point. pass@1 is the production signal: it is the chance the very first suggestion works, which is what a user of an autocomplete or a single-shot assistant experiences. Larger k, like pass@10 or pass@100, is an oracle bound. It describes the ceiling reachable if a perfect re-ranker or verifier could pick the best of k attempts. The word oracle is doing real work here, because in production nobody magically knows which of k samples is the correct one without running tests, and the user-facing tests may differ from the eval suite.
Reporting the curve across several k values is far more informative than any single point. The level of pass@1 tells you deployable quality. The gap between pass@1 and pass@100 tells you how much headroom a verifier, a test-time search, or a re-ranker could recover. Two models with the same pass@100 but different pass@1 are very different products. The first might be a strong base model you ship as-is; the second might be a high-diversity sampler that only shines once you bolt on a verifier to harvest its lucky draws.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's Codex paper introduced the unbiased pass@k estimator and reported it on HumanEval with n=200 samples per problem.
- HumanEval and MBPP, the standard code benchmarks, are scored with the unbiased pass@k estimator in 2026 model cards.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the plug-in estimator 1 minus (1 minus p) to the k biased even though the rate estimate is unbiased?
Invoke Jensen's inequality: pass@k is a nonlinear function of p, so the expectation of the function does not equal the function of the expectation. The Codex estimator averages the per-problem combinatorial probability instead, which is exactly unbiased.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Estimating pass@k by drawing k samples once and checking if any pass. That single Bernoulli trial is high-variance, so two models can swap ranks just from sampling luck.
60 second bullets to scan on the way to the call.
Definition of pass@k as probability at least one of k samples passes
Why pass@1 is the production signal and high k is an oracle bound
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.