What does pass@k measure and how is it computed to reduce variance?

Flashcard·Medium·4.0 · 0·~30s·Relevant atOpenAI

Attempt it

pass@k is the probability that at least one of k generated code samples passes unit tests. The unbiased estimator avoids high variance by generating n≥k samples total and computing: pass@k = 1 − C(n−c, k)/C(n, k), where c is the number of passing samples. This uses all n samples rather than randomly picking k, giving a lower-variance estimate than checking a single random draw of k samples.

Your answer

Quick reference

TL;DR

pass@k is the chance at least one of k code samples passes the unit tests. The unbiased estimator generates n more than k samples, counts c passes, and computes the probability in closed form for low variance.

Memory aid

Easy to grasp

Imagine a student who gets several tries to solve a puzzle, and they pass if any single attempt works. pass@k asks: with k tries, what is the chance at least one succeeds? You could just hand them k attempts and see, but that is noisy. A lucky or unlucky batch swings the number a lot. So instead you let them make many attempts, count how many actually worked, and then do the math for how likely k random tries would include a winner. Using every attempt you collected gives a far steadier number than judging from one small handful. Same idea code-eval teams use to grade a model on programming problems.

Key concepts

pass@k Code evaluation metrics Unbiased estimator Execution-based eval

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define pass@k as best of k functional correctness, derive the oversampling unbiased estimator, cover numerical stability, temperature interaction, and what the metric cannot see.

Code example

python

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    # n = samples generated, c = samples that passed, k = budget
    if n - c < k:
        return 1.0
    # 1 - product form of C(n-c, k) / C(n, k), numerically stable
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

Real-world examples

Real products, models, and research that use this idea.

HumanEval, the OpenAI Codex benchmark, defines pass@k with the unbiased estimator and remains a default code-eval metric in 2026.
MBPP and HumanEval+ report pass@1 and pass@10 using the same oversampling estimator over hundreds of samples per problem.
BigCode's evaluation harness and the EvalPlus toolkit compute pass@k in log space for models like Llama 4 and DeepSeek V4.
LiveCodeBench uses execution-based pass@k on fresh contest problems to reduce train-eval contamination for frontier LLMs.
SWE-bench reports a resolved rate that is conceptually pass@1 on real GitHub issues verified by the repository's own test suite.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the closed-form estimator lower variance than computing pass@k from a single draw of k samples?

QHow does sampling temperature change the optimal value when you target pass@1 versus pass@100?

QHow would you make pass@k robust when the unit tests themselves are weak or flaky?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Generating exactly k samples and reporting the raw hit rate. That estimate has high variance. Generate n more than k, count passes, and use the closed-form estimator instead.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.