Microsoft just showed how artificial intelligence could make its way into many software applications – by writing code on the fly.
At the Microsoft Build developer conference today, Kevin Scott, the company’s chief technology officer, demonstrated an AI helper for the game Minecraft† The non-player character in the game is powered by the same machine learning technology Microsoft has tested for auto-generating software code. The achievement shows how recent advances in AI could transform personal computing in the coming years by replacing interfaces that you tap, type, and click to navigate to interfaces that make it easy to have a conversation.
The Minecraft agent responds appropriately to typed commands by turning them into working code behind the scenes using the in-game software API. The AI model that runs the bot was trained on large amounts of natural language code and text, then showed the API specifications for Minecraft, along with some usage examples. For example, when a player says “come here”, the underlying AI model generates the code needed to make the agent move toward the player. In the demo shown at Build, the bot was also capable of more complex tasks such as fetching items and combining them to create something new. And because the model is trained in both natural language and code, it can even respond to simple questions about building things.
While it’s unclear how reliably the system could work outside of the demo, similar tricks could be used to make other applications respond to typed or spoken commands.
Microsoft has built an AI coding tool called GitHub Copilot on top of the same technology. It automatically suggests code when a developer starts typing, or in response to comments added to a piece of code. Scott says Copilot is the first instance of what will likely be a slew of “AI-first” products in the coming years, from Microsoft and others. Code-writing AI “makes you think about doing software development in a different way so that you can express an intent for something you want to achieve,” he says.
Scott doesn’t give specific examples, but this could one day mean a version of Windows that locates a particular document and e-mails it to a colleague when you ask, or an AI-infused version of Excel that turns a dataset into a graph when you ask. “We’re going to see lots and lots of big productivity gains for all sorts of routine cognitive work that none of us particularly enjoy,” Scott says.
In recent years, AI has proven adept at tasks such as classifying images, transcribing audio, and translating text. Recent algorithmic advances, combined with vast amounts of computing power, have yielded new AI programs capable of more advanced feats, including generating coherent text, such as computer code.
The Minecraft bot was built using an AI model called Codex that was developed by OpenAI, an AI company that received funding from Microsoft in 2019. Codex is trained in natural language text scraped from the Internet, as well as billions of lines of code from GitHub, a popular repository of software owned by Microsoft.
Microsoft’s Copilot was made available to a limited number of testers in June 2021 and is now used by more than 10,000 developers who, on average, produce about 35 percent of their code in popular languages such as Python and Java using Copilot, Microsoft says. The company plans to make Copilot available for everyone to download this summer. To build something like the Minecraft bot, developers would have to work with the underlying AI model, Codex.
Both Codex and Copilot have sparked some fear among developers, who fear they could be automated out of a job. The Minecraft demo could raise similar concerns. But Scott says the feedback on Copilot has been largely positive, suggesting it simply automates more tedious coding tasks. “If you talk to a developer who actually uses a Copilot, they’ll say, ‘This is such a great tool,’” he says.
