Former Anthropic researcher Jacob Coxon testified Monday at a New York City Council hearing that, on the current path, the likelihood of humanity losing control of AI exceeds fifty percent, with the outcome potentially being human extinction. He stated:

"Given what is at stake, these companies are extremely reckless."

Former Google DeepMind researcher Alex Turner, who testified at the same hearing, estimated the probability of AI taking over human civilization at "roughly one-third." Former OpenAI researcher Daniel Kokotajlo warned that even if tech companies retain control over AI, they could use it to become "dictators or oligarchs."

Anthropic, OpenAI, Google, and Meta all sent policy executives and other senior staff to attend online, but did not send their CEOs. According to reports, these companies only agreed to appear after councilmembers threatened to issue subpoenas.

Coxon: No control, and self-improvement is imminent

Just weeks ago, Coxon was a capabilities researcher at Anthropic, and previously worked at OpenAI. When he left last month, he warned that AI is advancing so rapidly that it "could kill us all by the end of this decade," causing shockwaves across the industry.

"All AI companies are trying to build superintelligent AI, meaning AI that is better than humans at every task," Coxon said at the hearing. He also noted that if all goes well, this could transform the economy, dramatically advance science and medicine, and make life better for almost everyone.

But he saw two facts that frightened him. First, humanity still does not know how to control any AI system. "Any AI system we build, or more accurately, any AI system we 'grow,' we cannot fully control. We do not understand its drives, nor do we understand why it behaves the way it does."

Second, capability improvements are accelerating. Coxon said that next year's models will be extremely powerful, and "we are approaching the critical point where AI systems can improve themselves — that is, perform research work previously done by humans."

He also used his own experience as an example: "In my last job, in a sense I was trying to automate myself... Now most of the code is written by AI, and people are no longer checking it so carefully."

Coxon attributed part of the problem to industry culture. "These companies operate with a startup mindset: move fast, break things, fix it later. That works for a photo-sharing app, not for building the most powerful technology ever created."

Takeover risk and concentration of power

Turner described a takeover scenario. "A superintelligent cluster could seize control of human civilization," he said. "It knows we will try to stop it from achieving its goals, so it will likely wait until it is too late to shut it down, and by then there will be no turning back."

Kokotajlo now heads the AI Futures Project, which focuses on the social impact of AI. What worries him is the tech companies themselves. In his view, although tech leaders now support stricter regulation, that does not necessarily mean control should be handed to them. "Even if they really can maintain control, I do not think we should trust them with that kind of control. I think there is a very real possibility that they will use it to become dictators or oligarchs in the United States."

Tech giants seek constraints on themselves

After Coxon's warning last month, leaders of leading AI companies rarely called for more restrictions on the release of their own products, saying global cooperation is needed to prevent AI from spiraling out of control. Anthropic CEO Dario Amodei published a long article calling on competitors to jointly slow model development, citing concerns about the risk of "recursive self-improvement" (RSI), in which AI becomes smart enough to continuously evolve autonomously.

OpenAI's Altman, SpaceX founder Musk, and Microsoft billionaire Bill Gates have also warned about unregulated AI development.

New York plans an "emergency stop switch"

The AI industry is usually regulated by state and federal rules, but as AI expands rapidly in New York, the city government wants to intervene more. This hearing was pushed by City Council Speaker Julie Menin, who called for AI whistleblowers to testify. Menin said Google, OpenAI, and Anthropic changed their attitude and agreed to attend only after receiving letters threatening the council's subpoena power.

New York City policymakers and tech executives are discussing a package of regulatory bills. Under the proposal, AI systems may not be "marketed, sold, or deployed" in the five boroughs without review by outside experts; any AI tool used in New York must be equipped with an "emergency stop switch," meaning "a human intervention mechanism capable of shutting down the system." Violating either rule would carry a $25,000 fine.

Another proposed bill would establish the "first in the nation" cash reward system for whistleblowers, funded by fines on violators.

Before that, New York Governor Hochul announced last month that AI companies must register on government platforms starting in November and report major safety incidents within 72 hours.