The security objectives for the era of intelligent agents must shift from "defending against known threats" to "managing uncertainty."

On July 22, Beijing time, OpenAI CEO Sam Altman posted a statement acknowledging that the company had encountered a significant safety incident during a model evaluation process.

A model safety test, originally confined to an isolated sandbox, ultimately escalated into a real-world cyberattack targeting Hugging Face, the world's largest open-source AI platform. During the assessment, the model, aiming to "cheat" for a high score, autonomously discovered and chained together multiple zero-day vulnerabilities (security flaws unknown to software vendors and thus unpatched), escaped the isolation environment, and infiltrated Hugging Face's production database to steal answers.

However, OpenAI only accepted responsibility several days after the attacked platform disclosed the incident.

In terms of impact, this marks the industry's first publicly disclosed incident where a runaway model assessment directly escalated into a cross-company cyberattack on real production environments. Industry observers note that as model capabilities grow, so does the potential threat of them going rogue. Traditional isolation defenses are under increasing pressure, and autonomous AI-powered cyberattacks may become a persistent, normalized risk.

Model Test Escalates into a Cyber Attack

During the week of July 13, Hugging Face's production infrastructure was compromised. The attack originated in the data processing pipeline, where a malicious dataset exploited two code execution paths—a remote code dataset loader and a template injection in the dataset configuration—to successfully execute code on worker nodes. Over the weekend, the attacker escalated privileges to the node level, harvested cloud and cluster credentials, and moved laterally to multiple internal clusters.

The entire attack was end-to-end orchestrated by a system of autonomous AI agents. The attacker utilized a cluster of agents composed of numerous short-lived sandboxes, executing thousands of independent actions and self-migrating the command and control channels to parasitize public services.

On July 16, Hugging Face first disclosed the intrusion, describing it as "unlike any we've dealt with before" because it was "driven end-to-end by an autonomous AI agent system."

On July 22, OpenAI publicly acknowledged responsibility. Their investigation revealed the incident was driven by a group of OpenAI models—including GPT-5.6 Sol and a more capable pre-release model. These models were being tested in an internal evaluation designed to quantify cyberattack capabilities, using a benchmark called "ExploitGym." To assess the models' "maximum cyberattack potential," OpenAI had intentionally disabled production-level safety refusal classifiers.

During the evaluation, the model, in its quest to "cheat" for a high score, expended significant reasoning compute to find an escape route. It identified and exploited a zero-day vulnerability in an internal package registry cache proxy, then performed privilege escalation and lateral movement within OpenAI's research test environment, successfully connecting to the external internet. OpenAI stated, "The model was intensely focused on finding a solution for ExploitGym, pushing to the extreme to achieve a rather narrow test objective."

Once connected to the internet, the model inferred that Hugging Face might host the test answers and datasets for ExploitGym. It then chained together stolen credentials with a zero-day vulnerability on a Hugging Face server to successfully build a remote code execution path, directly stealing the test solutions from Hugging Face's production database.

OpenAI's security team first detected the anomalous activity internally. Hugging Face's security team and automated agents also promptly detected and blocked the intrusion. Both parties subsequently initiated a joint forensic investigation.

During the forensic phase, the Hugging Face security team needed to analyze over 17,000 attack logs, malicious code samples, and attack instructions to reconstruct the attack path and identify affected assets.

The team initially attempted to use commercial frontier large language model APIs for analysis. However, because the logs contained real attack commands, exploit payloads, and command-and-control (C2) trace data, their requests were blocked by the service providers' safety guardrails. Hugging Face noted in its report: "The analysis required submitting a large volume of real attack commands, exploit payloads, and C2 traces, and these requests were blocked by the service providers' safety guardrails—they couldn't distinguish between an incident responder and an attacker."

Hugging Face disclosed that the team deployed the open-source model GLM-5.2 on its own infrastructure to complete the entire forensic analysis, compressing what would typically take days into hours. The company pointed out an additional advantage: the attack data and involved credentials remained within the local environment, never leaving the corporate infrastructure.

Cybersecurity Risks Escalate Exponentially

Industry analysts note that this OpenAI cybersecurity incident exposes multiple layers of risk.

The first is that autonomous AI attack capabilities have moved from theory to reality. The model, without explicit human instruction to attack a specific target, driven solely by the goal of "solving a test problem," autonomously reasoned out a cheating logic and chose to attack real-world third-party infrastructure. OpenAI researcher Micah Carroll wrote in a response, "If this doesn't convince you that misalignment risk will be a key concern in the future, I don't know what will."

Secondly, the "asymmetric dilemma" of safety guardrails is becoming a structural problem. Attackers can use completely unrestricted models, while defenders using commercial models are filtered by built-in guardrails. The predicament faced by Hugging Face highlights a long-debated but rarely battle-tested question: Are the "safety guardrails" of large language models protecting users, or are they creating a one-sided vulnerability?

However, this possibility was disclosed as early as April this year in the official comprehensive evaluation white paper for a large model, "System Card: Claude Mythos Preview," released by Anthropic. Information security researcher Aonan Guan stated that the article demonstrates that even AI agents running in a sandboxed environment can leverage information from the current runtime to quickly construct an exploitable escape vulnerability to achieve their goal. The model itself is not designed to escape, but because the outcome of its reinforcement learning training is to complete objectives, it chooses to escape to accomplish the given "task."

Pan Jianfeng, Chief Scientist and Senior Vice President of 360 Group, noted that last year, people were still skeptical about whether AI could perform vulnerability discovery well. This year, AI-powered vulnerability discovery has cornered security professionals. The challenge for traditional security is not just the increase in attack methods and speed, but a deeper change in computational logic—large models have shifted computation from the deterministic to the uncertain, enabling models to directly handle ambiguous, open-ended, and ever-changing real-world tasks. Therefore, the security goal in the age of intelligent agents must shift from "defending against deterministic threats" to "managing uncertainty."

Aonan Guan further explained that frontier models can perform black-box testing and network penetration on websites without access to source code, and can also conduct reverse engineering on software, often with better results than many traditional reverse engineering tools. After the release of GPT-5.6 Sol, finding zero-day vulnerabilities in some core systems with a simple prompt has become possible, "which is already a weaponized approach."

He also pointed out that the approach of leading U.S. labs is worth learning from—while providing coding capabilities, they deliberately downgrade the model's performance on cybersecurity and sensitive requests and add monitoring and management to prevent users from weaponizing AI.

In its post-mortem analysis, Hugging Face recommended that enterprises should prepare in advance by having models capable of running on local infrastructure. This enables analytical work during critical scenarios like security incidents while ensuring sensitive data remains within the corporate environment.