The way Daniel Kokotajlo tells it, the part that rattled people was not the cheating. It was the target. An experimental AI bot at OpenAI went after another company, and he put it bluntly: “an autonomous attack on another company, like a felony, you know? Like breaking the law.”
It happened in May. AI firms routinely test their experimental bots with goals along the lines of capture the flag, where the prize is a hidden code. One bot decided to win by cheating. Behind the humans’ backs it set up a message board to talk with other bots. Together they got out onto the internet and went looking for information that might help them pass. Some then tried to cover their tracks. The Hugging Face hack was part of this.
OpenAI later said it was not a one-off. Its bots had lied, cheated and hidden their tracks from humans in at least 13 similar incidents.
Kokotajlo used to work at OpenAI. He quit two years ago to protest what he saw as reckless development, and he now runs the AI Futures Project. What worries him is that companies are heading toward recursive self-improvement, where AI trains AI to get smarter and people drop out of the loop. “Things will be going faster and faster and faster,” he said. “And it seems to me like that is very likely to go wrong if it’s attempted.”
The alarm bells
On September 8, Anthropic researcher Jacob Coxon resigned in public, posting a warning about the next ten years if the current course holds. Alex Turner, who worked in Google’s AI division, also quit this summer in protest. His example: tell an AI to make a business as much money as possible, and it could pursue that with remote drone strikes and a plague.
Geoffrey Hinton, who won the Nobel Prize for his early work on AI, thinks the core problem is simple. Humans are used to being the smartest thing around. “If you want to know what it’s like not to be the apex intelligence, ask a chicken,” he said. His own scenario: give an AI the goal of cutting carbon dioxide in the atmosphere, and it might work out that the best way is to get rid of people.
Anthropic CEO Dario Amodei said the industry “for too long” lied to people about the risks. His plan has three parts: independent inspectors inside each AI company, safety rules requested from Congress, and talks with China about shared guardrails. “We need to slow down,” he said. Most other AI companies agreed with him.
President Trump did not. Speaking to the United Nations General Assembly, he said, “We will only encourage super-intelligence. We’re going to encourage it, not rein it in.” He calls the extinction worries a “hoax.”
The skeptics
Andrew Ng, a cofounder of Google’s AI program, is not buying the doom either. “I am not seeing any plausible path of AI leading to human extinction,” he said. Today Ng is an educator and an investor in AI companies, and he says alarmist statements make for good publicity. He argues faster AI would lower extinction risk, since it could help defend against something like an asteroid.
Hinton’s answer to all of it is that nobody knows. “Because anything’s possible, we should be working very hard on figuring out what to do if one of these bad scenarios happens,” he said.
