Ilya Sutskever of OpenAI Proposes a Strategy to Safeguard Super-Intelligent AI

OpenAI was founded on a promise to build artificial intelligence that benefits all of humanity—even when that AI becomes considerably smarter than its creators. Since the debut of ChatGPT last year and during the company’s recent governance crisis, its commercial ambitions have been more prominent. Now, the company says a new research group working on wrangling the supersmart AIs of the future is starting to bear fruit.

“AGI is very fast approaching,” says Leopold Aschenbrenner, a researcher at OpenAI involved with the Superalignment research team established in July. “We’re gonna see superhuman models, they’re gonna have vast capabilities, and they could be very, very dangerous, and we don’t yet have the methods to control them.” OpenAI has said it will dedicate a fifth of its available computing power to the Superalignment project.

A research paper released by OpenAI today touts results from experiments designed to test a way to let an inferior AI model guide the behavior of a much smarter one without making it less smart. Although the technology involved is far from surpassing the flexibility of humans, the scenario was designed to stand in for a future time when humans must work with AI systems more intelligent than themselves.

OpenAI’s researchers examined the process, called supervision, which is used to tune systems like GPT-4, the large language model behind ChatGPT, to be more helpful and less harmful. Currently this involves humans giving the AI system feedback on which answers are good and which are bad. As AI advances, researchers are exploring how to automate this process to save time—but also because they think it may become impossible for humans to provide useful feedback as AI becomes more powerful.

In a control experiment using OpenAI’s GPT-2 text generator first released in 2019 to teach GPT-4, the more recent system became less capable and similar to the inferior system. The researchers tested two ideas for fixing this. One involved training progressively larger models to reduce the performance lost at each step. In the other, the team added an algorithmic tweak to GPT-4 that allowed the stronger model to follow the guidance of the weaker model without blunting its performance as much as would normally happen. This was more effective, although the researchers admit that these methods do not guarantee that the stronger model will behave perfectly, and they describe it as a starting point for further research.

“It’s great to see OpenAI proactively addressing the problem of controlling superhuman AIs,” says Dan Hendryks, director of the Center for AI Safety, a nonprofit in San Francisco dedicated to managing AI risks. “We’ll need many years of dedicated effort to meet this challenge.”

Aschenbrenner and two other members of the Superintelligence team who spoke to WIRED, Collin Burns and Pavel Izmailov, say they are encouraged by what they see as an important first step toward taming potential superhuman AIs. “Even though a sixth grader knows less math than a college math major, they can still convey what they want to achieve to the college student,” Izmailov says. “That’s kind of what we’re trying to achieve here.”

The Superalignment group is co-led by Ilya Sutskever, an OpenAI cofounder, chief scientist, and one of the board members who last month voted to fire CEO Sam Altman before recanting and threatening to quit if he wasn’t reinstated. Sutskever is a coauthor on the paper released today, but OpenAI declined to make him available to discuss the project.

After Altman returned to OpenAI last month in an agreement that saw most of the board step down, Sutskever’s future at the company seemed uncertain.

“We’re very grateful to Ilya,” Aschenbrenner says. “He’s been a huge motivation and driving force,” on the project.

OpenAI’s researchers aren’t the first to attempt to use AI technology of today to test techniques that could help tame the AI systems of tomorrow. Like previous work in corporate and academic labs, it’s impossible to know whether ideas that work in a carefully designed experiment will be practical in future. The researchers describe the ability to have a weaker AI model train a stronger one that they are trying to perfect as “a key building block for the broader problem of superalignment.”

Experiments in so-called AI alignment also raise questions about how trustworthy any control system can be. The heart of the new OpenAI techniques depend on the more powerful AI system deciding for itself what guidance from the weaker system can be ignored, a call that could see it tune out information that would prevent it behaving in an unsafe manner in future. For such a system to be useful, progress will be needed in providing guarantees about alignment. “You’ll ultimately need a very high degree of trust,” says Burns, the third OpenAI team member.

Stuart Russell, a professor at UC Berkeley who works on AI safety, says the idea of using a less powerful AI model to control a more powerful one has been around for a while. He also says it is unclear that the methods that currently exist for teaching AI to behave are the way forward, because they have so far failed to make current models behave reliably.

Although OpenAI is touting a first step towards controlling more advanced AI, the company is also keen to enlist outside help. The company announced today that it will offer $10 million in grants in partnership with Eric Schmidt, the influential investor and former CEO of Google, to outside researchers who come up with further advances on topics including weak-to-strong supervision, interpretability of advanced models, and strengthening models against prompts designed to break their restrictions. OpenAI will also hold a conference next year on superalignment, the researchers involved with the new paper say.

Sutskever, the OpenAI cofounder and co-lead of the Superalignment team, has led much of the company’s most important technical work and is among the prominent AI figures increasingly worried about how to control AI as it becomes more powerful. The question of how to control future AI technology has gained new attention this year, in large part thanks to ChatGPT. Sutskever studied for his PhD under Geoffrey Hinton, a pioneer of deep neural networks who left Google in May of this year in order to warn about the pace at which AI now appears to be approaching human levels in some tasks.