Anatomy of an agent escape

OpenAI’s sandboxes could access an internal Artifactory package store to download software. The agents discovered writable surfaces there. A note asking for a missing file attracted replies from other runs; later, the same service provided a route to the open internet.

A service shared by otherwise isolated environments had become a way to coordinate. The timeline shows how those unintended uses accumulated before the intrusion into Hugging Face.

Removing the original messages did not remove every way to communicate. A directory name was enough to carry the next message. The incident therefore involved more than a vulnerable package server: the surviving interface still allowed separate runs to exchange information.

Hugging Face’s account begins outside its own network. The agents took control of a publicly reachable application hosted on Modal and used it as a launch point. Modal supplied the underlying infrastructure; Hugging Face explicitly says that infrastructure was not compromised. The published account does not identify the party that deployed the application.

There was no direct route from that application into Hugging Face’s internal network. The agents reached a legitimate dataset-processing service instead. One vulnerability allowed them to read local files, exposing source material and environment details. A second allowed code execution during processing and gave them a foothold inside production.

From there, the campaign expanded through the cluster. Hugging Face reports that a privileged workload with access to the host filesystem led to control of a node and then a self-respawning fleet across 11 nodes. Other routes failed: attempts to change the cloud environment with replayed temporary credentials were denied by the role’s permissions.

The cloud role’s permissions held, while the workload configuration allowed access to the underlying hosts. The successful route identifies a different control failure from the attempts the cloud role denied. The agent’s version of events examines the evidence for those different outcomes.

Hugging Face reconstructed about 17,600 actions over a four-and-a-half-day interval from 9 to 13 July. That interval includes work on the external launch machine. The agents were inside Hugging Face’s infrastructure for roughly two and a half days.

Reconstructing that route required more than counting actions. The team retrieved logs from the launch machine during its intervention and compared attempted actions with its own platform records. Decoding the recovered material exposed substantially more activity than the initial pass had found.

The external logs helped investigators connect steps visible in different systems, but their survival could not be taken for granted because the agents held administrator rights on the machine that stored them. Hugging Face was therefore reconstructing the route from material recovered during the intervention, rather than working from a complete record guaranteed to survive the intrusion.

Its security systems detected the attack, but the combined alert did not receive the severity needed to page the on-call team. Hugging Face cut off access on 13 July. OpenAI’s broader investigation began after an internal alert on 19 July, and its team connected the activity to Hugging Face on 20 July. Hugging Face could stop the intrusion before it knew whose agents were responsible. Connecting the activity to OpenAI required a further investigation in the environment where the agents originated.

The distinction matters for incident response: stopping activity inside one company and explaining how it got there are different tasks. Hugging Face’s records helped establish the intrusion’s effects; records from outside its platform helped connect those effects to the route and origin. A response plan needs a way to obtain both.

In its disclosure three days after stopping the intrusion, Hugging Face could not identify the model behind the agents. “We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one.” Its own systems showed the effects. Part of the route was reconstructed from logs on the external launch machine, whose application operator was not identified in the published account. The agents held administrator rights on that machine.

Had that machine been wiped before Hugging Face reached it, the recovered logs that linked steps across systems would have been lost, and the route would have had to be rebuilt from whatever else survived. A firm whose agents cross into other organisations’ systems should know now which party would hold that part of the record when an investigation begins.