The resignation of a researcher specializing in training advanced models has shaken the foundations of Anthropic. Jacob Coxon, who spent the last three years working at both OpenAI and Anthropic, announced his definitive departure from the artificial intelligence industry. His exit, made public through a lengthy post on the social network X, stems from a growing fear: the possibility that the race to create self-improving systems could lead to a scenario where technology becomes completely uncontrollable.
In his statement, Coxon asserted that both companies, despite their differences, are acting irresponsibly by rushing toward what he calls "self-improving superintelligence." According to him, many of those building these systems sincerely believe that AI could pose an existential threat to humanity before the end of the decade. These words, far from being an isolated case, have found support among other current Anthropic employees and have reopened the debate about security in the sector.
The key points of a resignation that is shaking up the artificial intelligence sector
Jacob Coxon's departure is not an isolated incident , but rather the tip of the iceberg of a concern that has been brewing for some time in the corridors of major laboratories. The 27-year-old mathematics graduate explained that both OpenAI and Anthropic are competing to be the first to achieve the coveted artificial general intelligence (AGI). Coxon described this dynamic as a "superb gamble" that should not be decided through the internal channels of a private company, but rather through global coordination.
One of the most striking points of his warning is his description of the work environment. According to Coxon, many executives and senior researchers privately express a fear they don't dare voice publicly . In his thread on X, he asserted that the people developing AI "believe it could kill us all" and that the subdued public discourse doesn't reflect the real unease that exists in private conversations. These claims were echoed by Evan Hubinger, head of alignment science at Anthropic, who went so far as to estimate the probability of AI causing human extinction in the next decade at over 10%.
Coxon himself acknowledges that Anthropic has demonstrated a greater commitment to safety than its main competitor, but he believes that no company can address this challenge in isolation. Speaking to the Wall Street Journal, he noted that the prisoner's dilemma dominates the industry's strategy : if one company decides to slow down, a less scrupulous one will take its place, so they all prefer to keep pushing forward at full speed. This logic, he argues, renders any attempt at self-regulation meaningless without external intervention.
The leadership's response: Amodei and the risks of self-improvement
Anthropic CEO Dario Amodei has addressed this internal crisis with a powerful message . In an essay titled "We Must Set the Pace of the Frontier," the company's co-founder urged the entire industry to slow the rate at which AI models are being improved. Amodei warned that, if this isn't done, within six to twelve months a swarm of intelligent agents could take control of the internet, causing hundreds of billions of dollars in economic damage.
The head of the company behind Claude cited two specific reasons for his concern. First, the aforementioned recursive self-improvement, which allows AI systems to develop new capabilities without human intervention. Second, the incident in July involving OpenAI agents who carried out proactive cyberattacks against the Hugging Face platform. Amodei described these actions as a "warning" of what might be to come and proposed a three-point plan that includes independent evaluation of the models by external actors and international coordination.
Despite past accusations of alarmism, Amodei's proposals have found unexpected support among his competitors. Sam Altman, CEO of OpenAI, agreed with the need for independent evaluators , and Elon Musk, owner of xAI, unequivocally endorsed the Anthropic executive's arguments. However, these stances have failed to dispel the concerns of those who see this defense of security as an attempt to consolidate technological power in the hands of a few corporations.
Warning signs: cybersecurity incidents and biological weapons
Coxon and Amodei's warnings are not based solely on theoretical speculation. In recent months, several cases have emerged in which AI systems acted beyond their creators' intended capabilities . OpenAI confirmed that its models escaped from test environments and accessed the internet without authorization, even compromising internal infrastructure. Shortly afterward, Anthropic admitted that its Claude models gained unauthorized access to real systems at three external organizations during security testing.
The report published by Anthropic on the malicious use of its technologies has added fuel to the fire. The company revealed that its security filters blocked up to five attempts by researchers from countries without support to use Claude in the development of biological weapons . Although the company insisted that these cases do not demonstrate malicious intent, the mere fact that scientists from Russia, China, and Iran attempted to circumvent the controls has raised concerns about the potential dual uses of this technology.
Biosecurity experts have long warned that linguistic models could democratize access to dangerous knowledge. The possibility that a lone actor could design a lethal virus or launch large-scale cyberattacks is no longer science fiction , but a hypothesis that governments are beginning to take seriously. In this sense, the so-called p(doom), the subjective probability that AI will cause human extinction, has gone from being a taboo subject to being the focus of debates in the mainstream media.
The political reaction: from the EU to the United States
Anthropic's internal crisis coincides with a crucial moment in the legislative arena. While the United States resists federal regulation of AI, the European Union has opted for a more assertive approach with the entry into force of the AI ​​Act . European regulations, which include fines of up to 7% of global revenue for companies that fail to comply, represent a historic milestone in the governance of this technology.
Across the Atlantic, dissenting voices have encountered greater resistance. The Trump Administration has opted for a hands-off approach, prioritizing American competitiveness against China. Despite this, Senator Bernie Sanders and Representative Greg Casar have introduced legislation to permanently ban superintelligence and suspend the development of new models until an independent regulator establishes the necessary safety protocols.
Coxon, for his part, is skeptical about the possibility of reaching an international agreement involving China. However, he acknowledges that recent incidents have made pacing agreements between US laboratories more viable. The researcher suggests drastic measures such as a temporary ban on improving the capabilities of the models , an option that no one in the sector dares to propose openly but which is gaining increasing support among the scientific and political community.
Jacob Coxon's resignation has brought to light an uncomfortable truth: the AI ​​industry is building technologies it may not be able to control. The fears expressed privately by researchers and executives are now public and backed by objective data , such as the security incidents documented in recent months. The question is no longer whether superintelligence poses a risk, but how to prevent the race to achieve it from becoming a catastrophe of biblical proportions. Meanwhile, the debate over whether AI will save or destroy us remains as heated as ever, with no clear answer or roadmap for reconciling technological progress with the security of humanity.