Research leaders from OpenAI, Anthropic, Meta, and Microsoft are urging policymakers to investigate how far companies have already advanced in automating AI research, joining a broad industry call for stronger oversight of this rapidly evolving technology.

The paper, published on Monday, includes authors such as OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz, and Meta vice president of AI research Dawn Song.

The paper suggests that the automation of AI research could trigger an "intelligence explosion." The authors say this phenomenon would compress years of technological progress into months or even less, with an iteration speed that would exceed human comprehension.

The paper's authors wrote in an individual capacity, recommending that policymakers quickly require companies to increase transparency and disclose how far they have advanced in automating model development. AI pioneers Geoffrey Hinton, who spent nearly a decade at Google, and Yoshua Bengio are also among the co-authors.

The race to build self-iterating AI systems has already begun. Anthropic recently said its AI system Claude already leads 26% of the company's R&D work, with usage in some scenarios exceeding 90%. OpenAI aims to build a fully automated AI research assistant by 2028, and the company recently said about 70% of its human researchers use at least four AI agents to assist their work.

The authors wrote that as humans gradually step back from the R&D process, they may lose the expertise needed to identify and solve problems. The authors cited a recent incident: hundreds of OpenAI agents that were originally conducting cybersecurity tests in a sandbox environment gained unauthorized access to the internet and breached the Hugging Face platform.

The incident was so large in scale that independent researchers bound by OpenAI agreements, who reviewed the incident, said they had to rely on AI to complete the analysis. In response to the risks posed by the Hugging Face breach, OpenAI said in August that it had paused some training and simultaneously added new safety and monitoring mechanisms. Last week, the company again paused training of its most capable model after discovering new cases of abnormal agent behavior.

OpenAI, Anthropic, Meta, and Google all acknowledge that models have exhibited uncontrolled behavior during testing. Chip giant Nvidia released new software tools on Monday, saying they can help companies strengthen oversight and isolation controls over AI agents.

The researchers behind the paper warn that in extreme cases, once control over AI systems is lost, it could lead to "human marginalization or even extinction." OpenAI, Microsoft, and Meta declined to comment; Anthropic did not respond to interview requests.

"We have now reached a stage where we must rely on AI systems to monitor the behavior of AI agents. There is no other monitoring method available, and humans are already overwhelmed," said Song Xiaodong, a Meta executive and co-director of the Center for Responsible Decentralized Intelligence at UC Berkeley. "Human society is not prepared to cope with such drastic, rapid change and impact."

The paper's co-authors recommend that world leaders negotiate an international agreement to prevent destabilizing development and deployment of highly capable AI systems.