In June 2021, GitHub has announced Copilot, a type of autocomplete for computer code powered by OpenAI’s text generation technology. It provided a first glimpse of the impressive potential of generative artificial intelligence to automate valuable work. Two years later, Copilot is one of the most mature examples of how the technology can take over tasks that previously had to be done by hand.
This week, Github released a report, based on data from nearly a million programmers who pay to use Copilot, that shows just how transformational generative AI coding has become. On average, they accepted the AI assistant’s suggestions about 30 percent of the time, suggesting that the system is remarkably good at predicting useful code.
The striking graph above shows how users tend to accept more suggestions from Copilot the more months they spend on the tool. The report also concludes that AI-enhanced coders see their productivity increase over time, based on the fact that a previous Copilot study reported a correlation between the number of suggestions accepted and a programmer’s productivity. GitHub’s new report says the biggest productivity gains were seen among less experienced developers.
At first glance, this is an impressive picture of a new technology that is quickly proving its worth. Any technology that increases productivity and enhances the capabilities of less skilled workers can be a boon to both individuals and the wider economy. GitHub continues with some back-of-the-envelope speculation, estimating that AI coding could increase global GDP by $1.5 trillion by 2030.
But the GitHub graph showing programmers bonding with Copilot reminded me of another study I heard about recently, while talking to Talia Ringer, a professor at the University of Illinois at Urbana-Champaign, about the relationship of coders with tools like Copilot.
Late last year, a team from Stanford University posted a research paper looking at how using a code-generating AI assistant they built affects the quality of the code people produce. The researchers found that programmers who were given AI suggestions tended to include more bugs in their final code, but those with access to the tool tended to believe that their code more Certainly. “There are likely both benefits and risks associated with coding in conjunction with AI,” says Ringer. “More code is not better code.”
Considering the nature of programming, that finding is not surprising. As Clive Thompson wrote in a 2022 WIRED feature, Copilot may seem miraculous, but its suggestions are based on patterns in other programmers’ work, which may be flawed. These guesses can create bugs that are fiendishly hard to spot, especially if you’re enchanted by how good the tool often is.
