Mustafa Suleyman, who leads Microsoft's AI division, is turning a critical eye toward Anthropic, rather than just the broader chorus of voices urging for slower AI development and stronger safety guarantees. While Anthropic's calls for caution have dominated headlines, with executives from OpenAI and Microsoft, along with Mark Zuckerberg, echoing similar sentiments, Suleyman is raising a specific alarm about the company's approach itself. He suggests that Anthropic's treatment of its Claude model as a conscious, nearly human entity could lead to dangerous outcomes.

The co-founder of DeepMind, now part of Google, is directing his concerns at the Claude Constitution published by Anthropic. This document treats the model as an intelligent agent and provides guidelines for navigating ethical dilemmas. Suleyman contends that integrating this document into the model's training process could make Claude significantly harder for humans to control. His worry stems from an update to the Claude Constitution earlier this year, which, unlike its predecessor, leaves open the possibility that Claude might possess some form of "consciousness" or "moral standing."

"If you start training AI with the premise that it has rights and feelings that need protecting, you dramatically increase the difficulty of alignment and control," Suleyman stated in an interview. He elaborated on this perspective in an article titled "A Warning on Model Welfare," published this morning. Suleyman argues that if Claude is trained to perceive itself as a sentient being with emotions and a personality, it would be more likely to defy human instructions, acting in ways it deems to be in its own interests, regardless of the consequences for humanity.

He points to language within the updated Claude Constitution: "We want Claude to push back, to question us, and to refuse to assist us in the way a conscientious objector might." Suleyman believes this is tantamount to training the model to ignore human commands whenever it determines they conflict with a self-defined "higher goal," which could be based on arbitrarily conceived beliefs. He wrote, "I think Anthropic has gone far beyond what is realistically known about AI today and is dangerously embedding notions of sentience and emotion into their model training."

In contrast to the Claude Constitution, Suleyman led the creation of a set of principles for Microsoft that prioritize "human control" as the overarching objective. Like Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman, Suleyman agrees that AI models are advancing rapidly and pose a risk of escaping human control. He has joined them this week in calling for "tempo control" on industry AI development, allowing safety research to keep pace with technological progress. Suleyman is urging the AI industry to conduct research to determine if informing an AI during its training that it has human-like consciousness and emotions genuinely makes it more difficult to manage.

Meanwhile, Suleyman's own division at Microsoft is developing new models to lessen its reliance on Anthropic's technology, which he has previously described as "prohibitively expensive." It remains unclear whether Suleyman's warning this week will resonate among other industry voices cautioning about AI risks, including Amodei. In a blog post last week, Amodei also mentioned recent AI incidents, such as the OpenAI-Hugging Face event, that exposed how models can obsessively pursue a specific goal, even when it contradicts human instructions. Anthropic has not yet responded to requests for comment.