A controlled study measured Copilot's effect on one task

A product launch and a separate controlled study

GitHub took Copilot out of technical preview and made it generally available to individual developers on 21 June 2022, at $10 a month or $100 a year, free for verified students and for maintainers of popular open-source projects. The announcement's own productivity claim was a usage statistic, not an experiment: in files where Copilot was enabled, the post reported that nearly 40% of code was being written by Copilot in popular languages such as Python, a measure of how much suggested code developers accepted, not of how much faster or better the resulting software was.

What the controlled study actually measured

A separate study by Peng, Kalliamvakou, Cihon and Demirer, submitted to arXiv on 13 February 2023, ran a controlled experiment: recruited developers were assigned to implement the same HTTP server in JavaScript, one group with Copilot access and a control group without it. The treatment group completed the task 55.8% faster. That is a specific, randomised comparison on one defined task in one language, and the paper itself reports heterogeneous effects across participants rather than a uniform speed-up, with the authors noting particular promise for people newer to software development. A 55.8% figure describes this task and this population; it is not a general multiplier for developer productivity applied elsewhere.

What sits behind both documents

Copilot's underlying capability had already been measured differently again, in the Codex paper that introduced the HumanEval benchmark: Codex solved 28.8% of HumanEval's docstring-to-function problems on a single attempt, rising to 70.2% when the model was allowed 100 samples per problem and the best one kept, and the paper states that a distinct production version of Codex powers GitHub Copilot. Three different documents therefore offer three different kinds of evidence about the same product: an acceptance-rate statistic from GitHub, a randomised task-completion-time result from an independent study, and a benchmark pass rate for the underlying model. None of the three substitutes for the others, and none establishes a single number for what a developer should expect Copilot to do for their own, differently shaped, task.

Questions to carry into your own evaluation

• Is a quoted productivity figure an acceptance rate, a task-completion time, or a benchmark score?
• Does the study's task and programming language resemble the work you actually do?
• Would the effect look the same for an experienced developer as for someone newer to the language?

An acceptance statistic, a controlled experiment and a benchmark score can all be true at once and still answer three different questions. Reading past the headline number to which of the three it is takes the comparison further than a single figure can.