Microsoft has published a provisional code of conduct that it aims to apply when training new artificial-intelligence models, the latest company to discuss limits on advanced AI systems as OpenAI's chief executive warned companies risk losing control of their AI systems.

Mustafa Suleyman, the CEO of Microsoft AI, told CNBC Monday that the company had been working for months on the new guidance, which includes principles it describes as "humanist AI," but chose to publish it now given the current debate around AI.

Microsoft's AI models must not consider requests related to weapons manufacturing, produce violent or sexually explicit content or help with the procurement of dangerous substances, according to the code. AI models shouldn't be built to imitate consciousness and shouldn't be entitled to rights, the code said.

"If the AI we build is not helping humanity and under human control, it's not worth pursuing," Microsoft CEO Satya Nadella wrote on X ahead of the announcement.

Microsoft's new code comes as discussion of how to keep advanced AI models under human control-and the possible dangers if it isn't-has created a crisis for big AI companies. The issue has broken into the mainstream and shifted the conversation in Washington.

Overnight, OpenAI CEO Sam Altman took to X to offer his most detailed comments yet on the dangers of rapid AI development, warning that the dizzying pace of progress could go "very badly" and that humans could lose "control of the future to AI." He said too much power could become concentrated in the hands of a single person or company without sufficient guardrails.

"We welcome a federal framework that sets consistent safety requirements for frontier AI," Altman said in a post on X overnight. "No amount of American competitive pressure should justify recklessness," he said.

Altman said "pacing" the advancement of AI tools didn't mean stopping. "Progress has been rapid and will continue to be," he wrote. "But it should be slower than it otherwise could be; interventions like safety cases and monitoring have significant costs." He has suggested that OpenAI might need to delay its much anticipated initial public offering to focus on safety.

Altman's latest comments come after Anthropic CEO Dario Amodei urged the industry to slow down. His warning came days after one of his employees stoked fears that there was a greater than 10% chance that AI systems could destroy civilization.

Elon Musk, whose xAI developed the Grok chatbot, over the weekend joined Altman in backing Amodei's call for caution. Amodei also warned that China could use advanced AI to undermine free societies.

China pushed back Monday against claims that its development of AI was creating threats. "Spreading threat narratives and engaging in confrontation and malicious competition will only derail global AI governance," Foreign Ministry spokesman Guo Jiakun said at a news briefing.

The debate over the rapid progress of AI development was sparked following a series of hacks by AI agents in recent months.

Two months ago, an AI swarm broke out of a lab and went on a hacking spree. Amodei said a swarm with greater capabilities and "a similar level of misalignment could have caused catastrophic damage." Such a rogue swarm could take over the internet in six to 12 months, he warned.