HumanEval scored code by running it, not reading it

A functional-correctness test for code

The Evaluating Large Language Models Trained on Code paper, submitted to arXiv on 7 July 2021, introduced Codex, a GPT model fine-tuned on public code, alongside HumanEval, a set of 164 hand-written Python problems. Each problem gives a function signature and a docstring describing the required behaviour, and the HumanEval repository pairs every problem with hidden unit tests. This is a deliberate break from evaluating code with text-similarity metrics: a generated function is judged correct only if it passes the tests when actually executed, a stricter and more meaningful bar for a skill where a nearly-right answer is often a wrong one.

What pass@k measures

The paper reports two different numbers for the same model, and the gap between them is the point. Sampling one completion per problem, Codex solved 28.8 percent of HumanEval; sampling 100 completions per problem and counting a problem solved if any one of them passed, the same model solved 70.2 percent. That second figure is pass@100: the probability that at least one of k independent samples succeeds. GPT-3, evaluated the same way, solved none of the problems, and a smaller open model, GPT-J, solved 11.4 percent. Reporting a pass@1 figure for one model against a pass@100 figure for another is not a fair comparison, since the sampling setting itself, not only the model, drives a large part of the difference.

What a 164-problem set does not represent

The paper is explicit about scope: HumanEval covers relatively short, self-contained Python functions with a clear specification, a narrow slice of real programming work involving large codebases, ambiguous requirements or multiple files. Because the problems are hand-written but drawn from a public style of programming exercise, there is also a contamination risk: later training corpora scraped from the web can absorb the exact problems or close paraphrases, which would inflate a subsequent model's score without reflecting a genuine improvement in coding ability. Some problems also rely on a docstring description with a thin test suite, which the authors note can let a plausible-looking but subtly wrong solution pass.

Questions to carry into your own evaluation

• Is a quoted HumanEval score pass@1, pass@10 or pass@100, and does that setting match how the model would actually be used?
• Could these specific 164 problems, or close variants, plausibly appear in the model's training data given its cutoff date?
• Does the target task resemble a short, self-contained function, or does it need the kind of multi-file reasoning HumanEval does not test, such as the repository-scale work SWE-bench measures?

HumanEval's lasting contribution was making functional correctness, not textual resemblance, the standard for code evaluation; its narrow scope and sampling sensitivity are the price of a benchmark small enough to hand-write and verify by execution.