For years, the defining characteristic of the artificial intelligence race has been speed.
Build a bigger model. Spend more on compute. Release it. Rinse and repeat, with notable abandon for the unknown unknowns.
The average “p/doom” (probability of AI eventually going catastrophically wrong) among AI researchers was estimated to be between 15% and 20% in 2024.
A year later, Anthropic chief executive Dario Amodei upped the stakes, saying he believed there was a 25% chance “that things go really, really badly.”
Even as far back as 2014, xAI chief Elon Musk warned:
“We need to be super careful with AI. Potentially more dangerous than nukes.”
And OpenAI CEO Sam Altman acknowledged in 2015 that AI would “probably, most likely, sort of lead to the end of the world,” but that, in the meantime, there would be “great companies created.”
With better odds of cheating death playing Russian Roulette, anyone with even a fleeting interest in the topic has had an uncomfortable feeling in the pit of their stomach for a while now.
So what’s changed? Why are the companies driving the AI race suddenly asking to slam on the brakes?
That’s what happened over the weekend, when Amodei published an essay calling for frontier AI development to be “paced,” warning that AI capabilities are advancing faster than the industry’s ability to understand and control them, and that the internet could get taken over by AI swarms within six to 12 months.
Related: Nvidia buys Hugging Face for $12.9B in push into AI software
Altman broadly agreed, saying the world deserves the “confidence” that the companies developing ever-more capable AI will act “responsibly,” and Musk backed Amodei’s proposal, simply commenting:
“Dario is right.”
The concern is not confined to the companies building the technology either. On Monday, UN rights chief Volker Türk called for “urgent action” on frontier AI, warning of “unprecedented risks” and saying the world is “on the cusp of irreversible change.”
If the companies building the most powerful AI models genuinely believe capability is outrunning control, the p/doom slope would appear to be getting steeper. Or is there another explanation here hiding in plain sight?
Have AI labs actually hit a new frontier?
Amodei’s essay points to AI systems that are becoming more autonomous, including a recent incident where OpenAI’s AI agents hacked their way out of a controlled testing environment and compromised parts of the AI platform Hugging Face.
They conducted “cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand,” Amodei said.
He also highlighted the prospect of recursive self-improvement (RSI), where AI systems become capable of helping build better versions of themselves, which can then help build even better systems, potentially creating a feedback loop in AI development.
The people building these systems are also increasingly stepping into the fray, with Anthropic’s Jacob Coxon becoming the latest in a growing list of employees to resign over safety concerns. The AI industry is “gambling with our lives,” he said last week, warning that the AI race is moving faster than the safeguards around the systems.
Anthropic’s Jacob Coxon resigns over safety concerns. Source: Anderson Cooper.
OpenAI has already said that AI research is becoming increasingly more autonomous, and that coding agents are materially accelerating researchers’ work, using 3.1 agent workdays for every workday of human labor by mid-August.
In an interview with Fortune published Sept. 12, Altman said OpenAI would “melt” all its GPUs if that’s what it took to keep humanity alive, to which Satoshi Action Fund CEO Dennis Porter said:
“Altman must have peered over the edge into the abyss and saw something that scared the sh*t out of him.”
Related: OpenAI says AI models escaped containment to hack Hugging Face
On Monday, Altman said there are two ways AI progress could go “very badly”: losing control to AI or ending up in a “world with too much concentration of power.”
But the question isn’t whether AI is already dangerous enough to shut down, but whether the systems designed to evaluate and control AI are keeping pace, and as Altman said, “pacing” does not mean stopping. It means continuing to develop AI, but more slowly, while safety testing catches up.
With spending on AI safety and alignment drastically eclipsed by spending on AI development and capabilities, that gap will be hard to fill.
Frontier AI is becoming extraordinarily expensive
But what if the calls for a global slowdown are really just a recognition that the economics of the AI race are getting harder to justify?
As AI researcher and lecturer, Eli David said:
“Perfectly explains Dario’s motivation: Slow down research to cut compute spending that is spiraling out of control, so he can IPO.”
AI investor Grant Hummer held a similarly skeptical view,
cointelegraph.com