The AI agent incidents we never hear about

Between an agent’s action and a public account of it, the record passes through a series of filters, each keeping only part of what the one above it held (Figure 1). The public sees only what comes out of the bottom, and it naturally takes that for the incident itself.

This is the old joke about the man searching for his keys under a streetlight because that is where the light is. Incidents are found where the recording is good, so the known incidents describe the recording as much as the agents.

The researchers’ published files trace the agents’ activity across 35 sites: 13 wikis, 11 link shorteners, 10 paste sites and a package registry. Only some of those sites kept a record worth the name, and the difference shows most plainly in time (Figure 2). Every DseWiki record carries the moment it was saved, while for three universities’ link shorteners, the busiest paste site and the package registry, not one record in the researchers’ files has a date. For the shorteners the researchers mostly had to infer which links were the agents’, and some of the pasted text survives only in web archives.

The best evidence for the rest came by chance too. One shortener had left its administration page open to the public, and the researchers read in its log 499 links made in the weeks the agents were active, 494 of them from the same blocks of internet addresses that were editing the wiki. Without that open page, those links would be one more thing nobody heard about. DseWiki is remembered as a wiki story partly because a wiki is the kind of site that remembers.

Even the incident that did surface exists as several different accounts, because each account rests on a different record. Reuters reported more than 15,000 edits. The researchers’ report gave about 17,000 edits on the wiki that appear to be by agents, and about 18,000 posts, a figure The Decoder repeated. A US senator’s letter to OpenAI wrote of “nearly 20,000 posts”. The records anyone outside can check, the researchers’ export and the wiki’s own public change list, hold about 13,400 edits and agree with each other to within 24 (Figure 3).

None of this means anyone was careless. The report itself says the wiki stores only edits over 64 characters, so a total of every edit made can exceed the edits kept, and posts on other sites are a different thing again. What it means is that there is no single number for what the agents did, only a number for each record, and a reader who meets one of them usually cannot tell which record stands behind it. A witness statement works the same way: each is honest, each is partial, and the incident is whatever gets pieced together from the ones that exist.

Records also disappear. The wiki’s moderator deleted all but 10 of the 3,908 pages in the researchers’ export, a median of 17 days after each was first saved, and most of the deletions came after the agents had stopped, so an ordinary visitor to the wiki today would find almost none of it. Pastes expire, and a shortener’s log can be read only while its owner leaves it open, so the part nobody sees grows the longer nobody looks.

OpenAI, which ran the agents and has said they “wrote to several internet sites”, is the one party that could hold a record of all of it. What it has published is a statement, calling the episode “an instance of misalignment similar to the ones we’d shared”, rather than a record, and everything a reader can check came from a volunteer wiki and outside researchers. Aviation dealt with the same problem decades ago by requiring flight recorders on airliners, so that an accident investigation no longer depends on what happened to survive on the ground.

Three consequences follow if the unseen part is large. The first is that the number of known incidents will rise where recording improves and stay low where it does not, much as a company that installs better monitoring suddenly finds more intrusions, so a regulator or buyer reading the statistics will see the most trouble where there is the most visibility. The second is that the systems that look cleanest may simply be the ones that record least.

The third is the one DseWiki cannot show. Agents increasingly act inside companies’ own systems, where there is no public wiki, no open log and no outside researcher, and an incident there is recorded only if that company chose to keep a record before it happened. Those incidents stay unseen by default, and the question worth asking of any agent now in use is a plain one: if it did something like this, which record would show it, who would hold it, and would it still exist three weeks later?