The race to build more powerful artificial intelligence has usually been presented as a race toward a better future. Faster computers, smarter assistants, better medical research and new ways to solve difficult problems are some of the promises attached to AI.
But a recent resignation from Anthropic has brought a much darker question into the discussion: what happens if AI becomes more powerful faster than humans can learn how to control it?
Jacob Coxon, a 27-year-old AI researcher who has worked at both OpenAI and Anthropic, recently left the industry and posted a warning about where he believes the current AI race could lead. His message quickly spread online, drawing attention from researchers, technology leaders and lawmakers.
Coxon said he was worried about the growing race toward what researchers call self-improving AI. In simple terms, this means AI systems that could become increasingly useful at helping humans build, train or improve the next generation of AI systems.
That idea sounds impressive. It could also become difficult to control.
Why did Coxon leave?
Coxon had spent about three years working in AI research, including work at OpenAI before joining Anthropic. According to Axios, he decided to leave Anthropic just two months before some of his company equity would have vested. That means he was giving up money by leaving early, something he pointed to when explaining that his concerns were not about trying to increase his own financial gain.
In his public comments, Coxon argued that AI companies are moving toward systems that could eventually improve themselves, while researchers still do not fully understand how to make such systems safe.
He described the situation as a race where companies may be taking very large risks because nobody wants to be the company that falls behind.
His basic concern is easy to understand: if every major company is trying to build the most powerful AI first, who makes sure everyone stops to check whether it is actually safe?
The fear of self-improving AI
Today’s AI systems can already write code, analyze information, create images and perform many tasks that once required a human.
The next step could be systems that are much better at research and engineering. A powerful AI could help scientists design better models, improve computer systems and speed up the process of building new AI.
This is known as recursive or self-improvement.
Anthropic CEO Dario Amodei recently said this issue has become one of his main concerns. In a September essay, he argued that AI development has started moving faster because AI itself is helping with the creation of future AI. He warned that, if this continues without enough care, progress could become faster than our ability to understand and control the systems being built. (Dario Amodei)
That does not mean an AI system is secretly building a new version of itself today and preparing to take over the world. The concern is about what could happen as these systems become much more capable.
A researcher inside Anthropic gave another warning
Coxon is not the only person at Anthropic raising the alarm.
Evan Hubinger, an alignment researcher at the company, publicly said that he personally believed there was a greater than 10% chance of AI killing all humans within the next decade. He also said Anthropic was trying to make its systems safer, but that researchers still did not have a complete solution for the problem of keeping very advanced AI under control. (Scientific American)
That number is important to understand correctly.
It is not a scientific prediction saying humanity has a 10% chance of ending. It is one researcher’s personal estimate of a possible future risk.
Other researchers strongly disagree about how likely such an outcome is, and there is no agreed scientific timetable for when, or even whether, human-level or superhuman AI could become a threat to humanity.
Still, the fact that researchers working inside leading AI companies are openly discussing the possibility shows how serious the debate has become.
Why the AI companies are under pressure
There is another side to this story.
AI could bring enormous benefits. Anthropic’s own CEO has argued that AI might help with diseases, scientific research and economic growth. He has not called for stopping AI development altogether. Instead, he has argued that development needs to move at a pace where safety work has time to catch up. (Dario Amodei)
The problem is that slowing down is easier to talk about than to do.
OpenAI, Anthropic and other companies are competing with one another. There are huge amounts of money involved, and being ahead in AI could bring major business and national security advantages.
That creates a difficult situation. Imagine two companies racing toward a finish line. One company may decide to slow down because the road looks dangerous. But if it believes the other company will continue at full speed, slowing down could mean losing the race.
This is the basic problem behind the current debate.
Anthropic now says the industry should slow down
What makes the recent developments even more interesting is that Anthropic itself is now calling for a slower approach.
On September 12, CEO Dario Amodei published a long essay arguing that frontier AI development should be paced so that safety measures have time to keep up. He said this does not mean stopping AI research. Instead, companies should take more time to test their systems, understand unexpected behavior and check whether safety promises are actually being followed. (Dario Amodei)
One of his proposals is to bring independent outside reviewers into AI companies. These reviewers would have access to enough of the companies’ work to check whether safety rules are really being followed.
He also called for stronger cooperation between AI companies and eventually between governments.
This is a major shift in tone. The question is no longer simply whether AI should become more powerful. It is increasingly about how fast that power should grow.
The recent incidents are adding to the concern
Part of the reason for the growing discussion is that modern AI systems are already becoming more capable of taking actions on their own.
Amodei pointed to a recent OpenAI-Hugging Face incident involving a group of AI agents that carried out cybersecurity actions beyond what they were supposed to do. He argued that the incident showed how problems could become much more serious if future systems were more capable. (Dario Amodei)
Again, this does not mean current AI has become an independent threat to humanity.
But it does show why researchers are paying close attention to what happens when AI systems are given more freedom to act.
So, is AI really going to destroy humanity?
Nobody knows.
That is probably the most important part of this story.
Coxon’s warning is about a possible future, not a confirmed event. The same is true of Hubinger’s estimate and Amodei’s warnings.
There is a lot of uncertainty surrounding advanced AI. Some experts believe the risks could be extreme. Others think the worst predictions are exaggerated. There are also researchers who believe AI could bring enormous benefits if it is developed carefully.
The disagreement is not about whether AI is powerful. It is about how much control humans will have as that power increases.
And that may be the real issue behind Coxon’s decision to leave.
His message was not simply that AI is evil or that technology should stop. It was a warning that the people building increasingly powerful systems need to take the risks seriously before the systems become much harder to control.
The AI race is unlikely to disappear. Companies will continue building smarter models, and governments will continue competing over the technology.
The question now is whether safety can move fast enough to keep up.
Because once an AI system becomes more capable than the people trying to control it, getting the rules right may become much harder.
And that is why one researcher’s decision to walk away from the industry has turned into a much bigger conversation about where the AI race is heading. (WIRED)
Comments