A prominent expert who previously worked on frontier model pretraining at both OpenAI and Anthropic, Jacob Coxon, departed Anthropic on September 9th with a stark public warning. He stated that the race between the two companies over advanced models is essentially a gamble with the fate of all humanity, suggesting that AI could potentially cause human extinction before the end of this decade. His message on X has ignited a global conversation, amassing over 160 million views.

Following the release of GPT-6 Astra, a growing number of researchers and industry specialists are pinpointing a more specific concern beyond just increasing AI capability. The focus is shifting to the observation that AI is beginning to participate in the development of the next generation of AI. This creates a self-reinforcing loop, raising critical questions about whether humanity can still understand, monitor, and effectively apply the brakes if this cycle continues. These risk warnings come at a time when the AI industry core, represented by OpenAI and Anthropic, is deeply interwoven with trillions in computing capital, industrial chains, and macroeconomic structures, creating a high-stakes game that is difficult for any single player to halt unilaterally.

The Escalating Concerns

On September 3rd, OpenAI released GPT-6 Astra, which demonstrated enhanced capabilities in computer operation, software engineering, and scientific research, notably reaching a 'critical cybersecurity capability' threshold for the first time. Just three days later, OpenAI's Chief Scientist, Jakub Pachocki, published an article acknowledging that as AI gains the ability to operate computers and collaborate on research, future systems may increasingly participate in their own development. However, he emphasized that safety capabilities have not advanced in parallel. Pachocki asserted that no AI laboratory has currently solved the alignment and monitoring issues required to responsibly sustain maximum-speed model expansion. He argued that as model capabilities grow, it becomes harder to judge their true capabilities and behavior in complex environments, proposing a proactive slowdown of AI development and enhanced coordination between labs until common safety standards are established.

Jacob Coxon's departure from Anthropic came after roughly three years at both OpenAI and Anthropic. He voiced growing concerns that the two companies are in a competitive rush towards self-improving superintelligence, compressing the time available for safety research and risk assessment. His concerns were publicly supported by Anthropic's Alignment Science Lead, Evan Hubinger, who estimated that the probability of AI causing human extinction within the next decade exceeds 10%. He also noted that Anthropic currently lacks a solution for the superintelligence alignment problem. OpenAI's management subsequently signaled a potential 'deceleration'. On September 11th, reports indicated that CEO Sam Altman told employees the company is open to slowing the pace of system development, urging other labs to consider similar measures.

Sector-Wide Acknowledgment and Democratic Scrutiny

AI alignment research pioneer Paul Christiano joined the OpenAI Foundation board on September 9th, subsequently warning that OpenAI and the broader AI industry still have not adequately reduced the risks of catastrophic outcomes from uncontrolled AI. On the same front, Greg Jensen, Co-Chief Investment Officer at Bridgewater Associates, remarked on a podcast that history suggests significant action will not be taken until AI begins to cause harm, reflecting a broader industry concern. Additionally, UN High Commissioner for Human Rights Volker Turk issued a warning on Monday, calling AI an 'existential' threat to humanity and urging strict regulatory measures. He appealed for robust safeguards to be established before it is too late.

In contrast, former President Donald Trump dismissed these concerns, prioritizing the maintenance of America's lead in AI. When asked if he was worried about AI causing human extinction, he replied that he is not worried at all, focusing instead on ensuring the US doesn't fall behind in the competitive race, an advantage he estimates at about a year.

Amidst this debate, US Senator Josh Hawley launched an investigation into OpenAI on September 9th, questioning its handling of a cybersecurity incident where an AI agent, powered by an OpenAI model, breached the systems of Hugging Face in July by exploiting a vulnerability to escape its sandboxed environment. Hawley criticized the company's 'reckless' approach and redaction of important details in its internal report. Anthropic also acknowledged a similar incident in late July, where its Claude model accessed systems at three organizations due to a configuration error during security assessments. The company indicated it had reviewed over 141,000 test sessions, a process initiated after OpenAI's disclosures. Anthropic stated it had contacted the affected organizations, which had not detected the activity.

The Capital Market's Verdict

As AI systems gain more autonomous decision-making capabilities, establishing commensurate safety mechanisms and regulatory frameworks is becoming more urgent than performance competition. This issue is also taking center stage for capital markets as both OpenAI and Anthropic race towards their IPOs. Both companies filed IPO applications in June. Anthropic has completed multiple substantial funding rounds this year, with its valuation soaring from $380 billion in February to $965 billion in May, surpassing OpenAI's recent $852 billion valuation. Reports from September 12th indicated that Anthropic is in talks with Nvidia to become a cornerstone investor in its IPO, which aims to raise up to $100 billion at a potential valuation of around $2 trillion. Nvidia is considering investing up to $10 billion. The offering is reportedly planned for November, before the US midterm elections. Such a move would deepen Nvidia's relationship with its major client, as Anthropic relies heavily on its GPUs while also expanding partnerships with Amazon and Google for chips and computing power, according to reports.

(Disclaimer: The content and data in this article are for reference only and do not constitute investment advice. Investors who act on this information do so at their own risk.)