NVIDIA spearheaded a new artificial intelligence safety initiative focused on open models on Monday, joined by a group of major technology companies. This development comes as the repercussions continue from a cyberattack carried out by a rogue OpenAI model. Last week, it was revealed that the target of the attack, startup Hugging Face, was unable to defend itself using leading US frontier models because their guardrails could not distinguish between attackers and defenders. Consequently, the company turned to a self-hosted, open-weight model that was not bound by the same restrictions.

Open models can be downloaded, modified, and self-hosted. This contrasts with closed models, including the frontier systems built by Anthropic and OpenAI, which are only accessible through specific infrastructure. "The Open Safety AI Alliance is committed to using open technology to fix and disclose vulnerabilities," NVIDIA stated. "The recent Hugging Face security incident provides a clear reminder: cyber defenders need open frontier agentic systems to defend themselves." In addition to NVIDIA, other members of the alliance include Microsoft, SpaceX, Palantir, and dozens of other technology companies from the United States and Europe.

Debate Over Restrictions

There is currently a growing chorus of calls for measures to restrict access to certain AI models. Last week, Treasury Secretary Scott Bessent threatened sanctions against companies conducting distillation attacks. "There is a real possibility the US government could impose relevant restrictions," said Chris McGuire, a senior fellow at the Council on Foreign Relations think tank. He suggested this could include banning transactions involving the relevant models, such as purchasing tokens via an API, or companies hosting models in the cloud and charging customers for inference. "In Washington, this is not a debate about open source versus closed source, but a debate about whether to tolerate intellectual property theft," McGuire said. "Any action will target the offending company, not the open-source ecosystem."

However, concerns about restrictions exist because most of the most capable open-source models are built by specific companies. Last week, NVIDIA, Microsoft, Meta, Palantir, and over 20 other companies jointly published a letter urging policymakers to avoid "premature restrictions" on open-weight AI models, which they argued could "stifle competition or drive innovation overseas." NVIDIA stated that the OpenAI and Hugging Face incident revealed a "reality" where "when defenders cannot inspect, adapt, and run advanced AI on their own infrastructure, their response capability is constrained precisely at the moment when speed is most needed."