Pass At K

Evaluating a code generation model on real world tasks beyond HumanEval: which metrics cover correctness, efficiency, and style?

HumanEval is the go to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.

Explain the pass@k metric and why the unbiased estimator matters for code gen evaluation

What does pass@k measure and how is it computed to reduce variance?