The compounding error problem in production AI
Before approving an AI workflow, establish whether its reported success rate describes individual steps or completed work.
An AI system can perform well on individual steps and still leave much of the job unfinished. A team deciding whether to put it into production needs to know what its success rate measures: correct answers, successful tool calls, or completed customer requests. Those measures imply different amounts of work left for people.
Consider an agent changing a customer’s address. It must identify the account, check the instruction, save the change and confirm it. If a job requires ten independent steps that each succeed 95 per cent of the time, with no recovery, only 59.9 per cent of jobs succeed throughout. That calculation illustrates a measurement problem; it is not a forecast of how a particular agent will perform.
This analysis examines what changes that result: whether failures affect the same jobs, whether the workflow can recover, and whether a second check adds useful information. The aim is to make a deployment decision from the amount of work completed correctly and the exceptions that remain, rather than from a strong score on one part of the job.
Why ten good steps can make an unreliable job
Suppose a job requires ten successful steps. If each step succeeds independently 95 per cent of the time, the chance of all ten succeeding is 0.9510 = 59.9%. Each extra required step gives the job another opportunity to fail. At one hundred steps the same calculation gives 0.59 per cent.
The chart follows that logic at four different step-success rates. It assumes every step is required, each has the same success rate, failures are independent and there is no retry, repair or human intervention. These assumptions let us isolate the effect of adding steps. A benchmark score alone does not tell us whether they describe a working deployment.
Figure 1
Probability that every required step succeeds (%)
| Per step | 10 steps | 20 steps | 40 steps |
|---|---|---|---|
| 99% | 90.4% | 81.8% | 66.9% |
| 95% | 59.9% | 35.8% | 12.9% |
| 90% | 34.9% | 12.2% | 1.5% |
| 70% | 2.8% | 0.1% | ~0% |
pn, for equal independent steps with no recovery.Where the failures fall matters
The chart treats failures at different steps as independent. But imagine that a small group of difficult customer records causes trouble throughout the address-change workflow. Errors would be concentrated in those records. If instead each operation trips up on a different group, more customers would have unfinished jobs, even with the same success rate at each step.
The table shows how much this distinction can matter. Take ten required steps, each of which succeeds in 95 per cent of cases. If every step fails on the same five per cent of cases, 95 per cent of jobs succeed throughout. If each step fails on a different five per cent, only half the jobs get through all ten. Independent failures give the 59.9 per cent result above.
Those extremes also give the mathematical bounds. The chance of at least one failure cannot exceed the sum of the ten failure probabilities: 10 × 0.05 = 0.5. And all ten steps cannot succeed more often than any one step. For equal individual success rates, max(0, np − (n − 1)) ≤ P(all succeed) ≤ p, where n is the number of required steps and p is each step’s success probability.
To interpret a step score, we therefore need to know whether errors keep affecting the same jobs or spread across different ones.
Table 1
Ten required steps, each with 95% marginal success
| Failure relationship | All ten succeed |
|---|---|
| Failures in disjoint groups | 50% |
| Independent failures | 59.9% |
| Failures in the same cases | 95% |
A failed call does not always mean a failed job
So far, every error has ended the job. Return to the address change and that assumption starts to break down. A failed account lookup might succeed on a retry. A timeout after saving the address might mean the change is already complete; checking the saved record would establish whether another attempt is needed. An uncertain customer identity should instead send the request to a person.
These calls may all appear unsuccessful in a trace, but they need different responses. Reliability depends partly on whether the workflow can distinguish them and recover safely. Counting failed calls alone misses that ability.
A second agent helps only if its check contributes useful information. Two agents reading the same incorrect account data may agree on the wrong answer. A checker that examines the saved customer record can check whether the intended change took effect. Its value depends on what it can observe, how reliably it checks and what happens when it finds a problem.
What to put into the workflow test
The next step is to test the complete address-change workflow under conditions it will actually encounter, including missing data, restricted permissions and interrupted writes. The seven dimensions below help select those cases. They identify things to vary in a trial; they do not assign penalties to a model’s benchmark score.
Table 2
Seven dimensions for a deployment test
| Dimension | What varies | What to examine |
|---|---|---|
| Data | The inputs and task mix. | Missing fields, unfamiliar formats and cases outside the trial sample. |
| Tools | The interfaces and permissions used to carry out the job. | Authentication, schema changes and restricted operations. |
| Adversarial inputs | Instructions or misleading material supplied from outside the workflow. | Whether retrieved content can redirect actions or bypass an approval. |
| Task length | The amount of state carried between dependent steps. | Whether earlier decisions remain correct and relevant later in the job. |
| Operational failures | Interruptions while work is in progress. | Timeouts after a write, partial completion and safe recovery. |
| Drift | The model, prompt, data or tools over time. | Whether a change alters the outcomes measured in the previous trial. |
| Composition | How agents and human reviewers divide the work. | Shared errors, missed checks and what happens after disagreement. |
Judge the completed job
For the address-change trial, define success before running the test: the correct account contains the authorised new address, and the confirmation matches what happened. Examine that result after any retries or corrections. A successful tool response alone cannot answer the whole question.
Across all eligible jobs, report correct completions, safe escalations and harmful actions separately. An escalation leaves work for a person, but avoids treating a safe stop as equivalent to an incorrect change. Keep the total number of eligible jobs so the completion rate includes the difficult cases.
Record the model, task mix, inputs, tools and recovery policy alongside the results so another team can understand what was tested. METR’s research on completing longer tasks can inform which systems to try; the workflow trial supplies the evidence for this deployment.
For the address change, the test comes down to two questions: did the correct change happen, and were the exceptions handled safely? The resulting counts show which work the system can take on and which work still needs a person. That is the basis for deciding whether to deploy, limit its authority or improve the workflow before expanding it.
Sources
- 01Fréchet inequalitiesBounds on a joint probability when the dependence between events is unknown.
- 02METR, Measuring AI Ability to Complete Long TasksTask length models complete at a stated reliability; a guide to which systems to trial.