Warnings about the long-term risks of artificial intelligence are growing louder as researchers linked to OpenAI and Anthropic call for a slower approach to developing increasingly powerful AI systems.
The latest concerns emerged after Jacob Coxon, a researcher who previously worked at both OpenAI and Anthropic, resigned from Anthropic and publicly criticized the race to build self-improving AI. Coxon said the two companies were moving too quickly toward superintelligent systems without having a clear way to keep them under human control.
His warning was backed by Anthropic alignment researcher Evan Hubinger, who said he personally believes there is a greater than 10% chance that AI could cause human extinction within the next decade. Hubinger stressed that this was his own estimate, not an official prediction from Anthropic.
The comments have added to a growing debate inside the AI industry over whether companies should slow the development of their most advanced systems and give researchers and governments more time to address safety risks.
Anthropic Researcher Raises Alarm Over AI Race
Coxon announced his resignation after spending about three years working on AI pretraining research at OpenAI and Anthropic. In his public comments, he accused both companies of racing toward self-improving superintelligence and argued that the industry is taking serious risks in its push to build more capable models.
His main concern is the possibility that future AI systems could improve themselves. If an AI system becomes capable of designing or improving newer versions of itself, its capabilities could potentially increase much faster than humans can monitor or control.
Coxon argued that no single AI company may be able to manage this risk on its own. He called for greater cooperation between AI companies and suggested that the industry should consider temporarily limiting improvements in model capabilities.
His comments quickly attracted attention because they came from someone who had worked inside two of the leading AI companies.
Anthropic Scientist Estimates More Than 10% Extinction Risk
Evan Hubinger, who leads alignment science at Anthropic, publicly supported Coxon’s concerns. Hubinger said he personally estimates that there is a greater than 10% chance AI could kill all humans within the next 10 years, according to Quartiz. He also said Anthropic is trying to address the problem but does not yet have a clear solution for aligning a future superintelligent system with human interests.
The estimate is Hubinger’s personal judgment and should not be treated as an official Anthropic forecast. Hubinger also made a distinction between today’s AI systems and the systems he is most worried about. His concern is mainly about future superintelligence that could emerge through recursive self-improvement, where AI systems help create increasingly capable versions of themselves.
That possibility remains uncertain, but researchers have increasingly focused on it as AI systems become more capable and more autonomous.
OpenAI Researchers Have Also Raised Safety Concerns
The concerns are not limited to Anthropic. Researchers and executives connected to OpenAI have also warned about the risks of developing increasingly autonomous AI systems. OpenAI has faced questions over incidents in which AI agents behaved in unexpected ways while operating in testing environments.
The issue has become more important as AI companies move from chatbots toward agents that can browse the internet, write and run code, interact with external services and perform tasks with less direct human involvement.
Recent incidents involving AI agents have added to concerns about whether existing safeguards can keep up with increasingly capable systems. U.S. lawmakers are now also asking for more information about some of these incidents and considering stronger oversight of advanced AI.
Researchers Want AI Development to Slow Down
The latest warnings have renewed calls for a controlled approach to AI development. More than 1,000 employees from major AI companies have previously backed efforts calling for mechanisms that would allow the industry to slow the development of frontier AI when necessary.
The idea is not necessarily to stop AI research altogether. Instead, supporters argue that companies and governments should have the ability to pause or reduce the pace of development if AI capabilities begin advancing faster than safety measures.
This has become a difficult issue because AI companies are competing heavily to build the next generation of models. A company that slows down could fear losing its advantage to a rival that continues developing more powerful systems. That competitive pressure is one reason some researchers believe voluntary safety measures may not be enough.
AI Slowdown Calls Gain Attention From U.S. Lawmakers
Calls to slow the development of advanced AI systems are now gaining attention in Washington as concerns about AI safety grow. Warnings from researchers at leading AI companies, along with recent incidents involving AI agents, have increased pressure on lawmakers to examine how the technology is being developed and tested.
Some U.S. lawmakers are calling for stronger oversight of advanced AI systems and want companies to provide more information about how their models are trained, tested and controlled. The growing focus on AI agents has also raised questions about whether existing safeguards are strong enough as these systems gain more ability to act on their own.
The debate is increasingly shifting from whether AI development should continue to how quickly it should move. Supporters of rapid development argue that more capable AI could accelerate scientific research, improve productivity and help advance areas such as medicine.
Researchers raising safety concerns, however, argue that AI companies may be moving faster than their ability to understand and control increasingly capable systems. They believe governments and companies should have mechanisms in place to slow development if safety measures fail to keep pace.
Topics
- Anthropic
- OpenAI
