What it takes for an AI agent incident to become public

Of the register’s 44 entries, 42 came from the developers and testers of the models themselves (Figure 1).

Businesses that deploy agents do have incidents of their own. In a survey of technology leaders by the software company Gravitee, about a third confirmed an agent security or privacy incident in the past year, at firms that monitor about half their agents. That is a broader category than METR’s, so the two cannot be compared directly; what the survey shows is that such incidents happen at firms whose records are not public.

The researchers traced the agents across 35 sites, mostly wikis, link shorteners (services that shorten web addresses) and paste sites (where anyone can post text publicly). Each kept a different fragment, in its own form (Figure 2). The wiki timestamped every edit it stored, while several others, including three university shorteners, left the researchers no dates at all.

Where the sites recorded little, the researchers had to infer which short links were the agents’, and some pasted text survives only in web archives. The exception was one shortener that had left its log of new links public: almost every link created while the agents were active came from the networks editing the wiki. Recorded evidence can also be lost. The wiki’s moderator deleted almost every page in the researchers’ copy, most within three weeks of being saved, and their text survives because the researchers had copied it.

OpenAI ran the agents and has said they “wrote to several internet sites”. Whatever logs it kept of its agents’ actions would be the only account covering every site from the agents’ side. It has released a statement calling the episode “an instance of misalignment similar to the ones we’d shared”, and the outside account rests on a volunteer wiki and outside researchers. The businesses in the survey are in the same position with their own incidents. Airliners must carry flight recorders so that investigators need not rely on what survives on the ground; agents have no equivalent.

Even when a scattered incident surfaces, each account rests on a different record. Reuters, the researchers’ report and a US senator’s letter to OpenAI put the count between 15,000 and 20,000, while the two records anyone can check hold about 13,400 edits and agree almost exactly (Table 1).

Part of the spread is explained: the report notes that the wiki stores only edits above a minimum length, and posts are a different unit. A reader who meets one of these numbers usually cannot tell which record it came from.

Agents increasingly act inside companies’ own systems, where there is no public wiki and no outside researcher. Some incidents there will surface through their effects (e.g. a customer’s complaint, or accounts that don’t reconcile), but knowing that something went wrong differs from reconstructing what the agent did.

An empty incident list therefore says little unless the evidence behind it would have revealed the failure in question. For any agent in use, that comes down to three questions: would the failure be recorded, could one party assemble the record, and would that party release it before the record is gone?