Anthropic researcher Jacob Coxon resigned from the AI company after accusing it and OpenAI of racing toward self-improving superintelligence without adequate safeguards.
Coxon spent the past three years working on AI pretraining research at OpenAI and Anthropic, and said on Tuesday that both companies are “gambling with our lives.” His warning quickly went viral, and attracted more than 70 million views on X.
“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon wrote.
His concerns were shared by Evan Hubinger, Anthropic’s alignment science lead, who said he believes there is a more than 10% chance AI could “kill all humans” in the next decade. Hubinger later clarified that he considers the risk from today’s models low, with his concern centered on future systems capable of improving themselves.
AI safety warnings are now coming from inside the labs
The timing makes Coxon’s resignation particularly interesting. OpenAI chief scientist Jakub Pachocki wrote on Sept. 6 that no AI lab has solved alignment and monitoring well enough to keep scaling at maximum speed for much longer. He said he expects voluntary slowdowns and wants governments to prioritize international coordination.
Blog post by Jakub Pachocki
Those comments follow a July cybersecurity incident that was disclosed by OpenAI in which internal research models circumvented isolation controls, exploited infrastructure vulnerabilities and reached systems belonging to Hugging Face. OpenAI called the episode a “warning shot” and said in August that its largest planned frontier reinforcement-learning run was on hold while it strengthened security and alignment safeguards.
Anthropic has acknowledged that AI is already accelerating AI development itself. The company says full recursive self-improvement — where a system autonomously designs and builds a more capable successor — has not been achieved and is not inevitable. However, Anthropic says its engineers now ship about eight times as much code per quarter as they did between 2021 and 2025.
Washington is starting to draw lines around frontier AI
Congress is also beginning to confront the issue. The bipartisan FRONTIER Act, which was introduced in July, would impose risk-based requirements on leading AI developers, including independent audits, incident reporting and safety frameworks.
A more aggressive proposal announced by Sen. Bernie Sanders and Rep. Greg Casar on Sept. 3 would permanently ban artificial superintelligence and temporarily pause advanced AI development until a federal regulator establishes safety rules.
Overall, the AI safety debate is becoming an argument among the people building the technology themselves over whether technical safeguards, corporate incentives and regulation can keep pace with advancing capabilities.
