Explain the pass@k metric and why the unbiased estimator matters for code gen evaluation

Attempt it

Explain the pass@k metric for code generation evaluation. Why is the naive estimator (randomly pick k samples and check if any pass) high-variance, and how does the unbiased estimator correct for this?

Free · 2 AI evals / day

Quick reference

TL;DR

pass@k is the chance at least one of k code samples passes the tests. Estimate it from n samples with the unbiased Codex formula, not a single random draw of k, which is high-variance.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a student who solves a tricky problem correctly about a third of the time. pass@k asks: if they get k attempts and we keep the best one, how often does at least one attempt work? You could measure this the lazy way: have them try exactly k times and see if any passed. But that's a coin flip with few tosses, so the number bounces around wildly between problems. The smarter way is to let them try many more times, count how many attempts succeeded overall, and do the arithmetic for what k tries would give. Using all those extra attempts makes the estimate steady instead of jumpy. It also reminds you that one lucky success out of many is not the same as a student who reliably gets it right.

Key concepts

pass@k Code evaluation metrics Unbiased estimator Evaluation variance

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define pass@k, motivate k=1 versus oracle k, show why the plug-in and single-k draw are biased and noisy, derive the closed-form estimator, and cover what pass@k hides.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's Codex paper introduced the unbiased pass@k estimator and reported it on HumanEval with n=200 samples per problem.
HumanEval and MBPP, the standard code benchmarks, are scored with the unbiased pass@k estimator in 2026 model cards.
Frontier model releases like GPT-5.5, Claude Opus 4.7, and DeepSeek V4 report pass@1 alongside higher-k oracle bounds on coding suites.
BigCodeBench and the EvalPlus harness implement the closed-form estimator directly rather than the naive random-k draw.
LiveCodeBench uses the same estimator to track contamination-resistant coding scores over time.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the plug-in estimator 1 minus (1 minus p) to the k biased even though the rate estimate is unbiased?

Sign in or go Pro to see the approach for this follow-up.

QHow would you compute the binomial ratio stably when n is 200 and k is 100?

Sign in or go Pro to see the approach for this follow-up.

QTwo models have equal pass@100 but very different pass@1. Which do you ship and why?

Sign in or go Pro to see the approach for this follow-up.

QWhat does pass@k miss about correctness even when it is high?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Estimating pass@k by drawing k samples once and checking if any pass. That single Bernoulli trial is high-variance, so two models can swap ranks just from sampling luck.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Chen et al., Evaluating Large Language Models Trained on Code