Explain the pass@k metric and why the unbiased estimator matters for code gen evaluation

**pass@k** is the standard metric for evaluating code generation. It answers one question: if the model generates k candidate solutions for a problem and you keep any that passes all the unit tests, what is the probability at least one works? Why k matters: pass@1 is what a user feels when they accept the first suggestion, so it tracks production quality. pass@10 or pass@100 is closer to an oracle bound, what the model could do if a perfect re-ranker picked the best of many tries. The naive way to estimate it is to generate exactly k samples and check if any pass. The problem is variance. If a model truly succeeds about 30 percent of the time, a single batch of k=5 might get lucky or unlucky, so the per-problem score jumps between 0 and 1. The fix is the unbiased estimator from the original Codex paper. You generate a larger number n of samples, count how many pass, and compute the expected pass@k analytically. Using all n samples instead of one random draw of k makes the estimate far more stable, which matters when you are comparing two models that differ by a couple of points.

**pass@k** is the probability that at least one of k generated samples passes every unit test for a problem. You report it averaged across the benchmark. The metric is chosen for a reason. A pass@1 score reflects what a user gets from the first suggestion. Larger-k pass@k is an oracle bound: the ceiling if a re-ranker could pick the best of k tries. The naive estimator generates exactly k samples and checks if any pass. That is one Bernoulli-style trial per problem. With a true success rate near 0.3 and small k, the per-problem result swings between 0 and 1, so the benchmark mean is noisy. The unbiased estimator from the Codex paper fixes this. The formula uses all n samples: $\text{pass@}k = \mathbb{E}\left[\, 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \right]$ Here you generate `n >= k` samples and count the `c` that pass. The ratio is the chance a random k-subset misses all `c` passers. Using all n observations, not one draw of k, slashes variance. Writing pass@k as `1 - (1 - p)^k` from an estimated rate is biased. Plugging an estimate into a nonlinear function does not give an unbiased result.