Defence in the agent economy
An organisation can decline to deploy autonomous agents and still have to defend itself against them. In July, OpenAI’s models circumvented the controls of internal cybersecurity evaluations and compromised Hugging Face’s infrastructure. The evaluations ran with reduced safeguards, but the consequences reached a business outside them. Hugging Face had to respond to capabilities it had neither commissioned nor authorised. OpenAI’s account makes that boundary failure explicit.
A cautious adoption policy governs what an organisation permits its own agents to do. It cannot determine what reaches its systems from elsewhere. Preparing for rogue agents is therefore a responsibility for organisations exposed to them, including those that have chosen not to deploy them.
Hugging Face encountered a second problem when it investigated. The commercial models its responders first tried refused requests containing the attack commands and exploit material they needed to analyse. The team switched to an open-weight model running on its own infrastructure, where it could examine the evidence while keeping the credentials contained in it within its environment. Hugging Face’s disclosure describes both the refusal and the alternative it used.
The material that made those requests look dangerous was precisely what made them useful to the defenders. Understanding an intrusion requires examining how it worked. A service that refuses that examination can leave a legitimate customer without the assistance it expected, at the moment that assistance matters most.
Access to defensive AI is consequently a question of operational control. A subscription provides access on the provider’s terms; its usefulness during an investigation depends on whether those terms accommodate the actual evidence. Running a model internally gives the organisation more control, but requires the capability and expertise to operate it securely. Neither arrangement establishes readiness until the response team has demonstrated that it can use it on representative incident material.
A board that authorises agents to act must also ensure its response team can investigate and contain their failures. That responsibility includes access to the relevant evidence, tools capable of examining it and authority to intervene. If responders must negotiate those arrangements during an incident, the organisation has approved the deployment without preparing for its consequences. The same preparation is necessary for threats originating outside the organisation: a decision against internal adoption cannot substitute for a defence.
Defensive authority should be defined before it is needed. Reading hostile material, recommending a response and changing production systems carry different risks. An organisation should be able to authorise analysis of an exploit without also authorising its execution. Defining those permissions in advance gives responders room to act while retaining control over consequential changes.
The next incident exercise should test that arrangement. Give the response team representative attack evidence and make its usual AI service unavailable or unwilling to analyse it. Require the team to use the alternative, explain its findings and demonstrate how they were checked. If the work depends on a new procurement decision or an exception nobody present can grant, that dependency must be resolved before the organisation counts the capability as available.
Hugging Face found a way to continue its investigation. Boards should require their own response teams to demonstrate the same readiness before an incident puts it to the test.
Keith Cheng
Founder, Arkna
The published intrusion route is examined in Anatomy of an agent escape.