The 99% task-success number is impressive, but the 98.8% residue rate is arguably the more important finding for anyone thinking about LLMs in production infrastructure.
It highlights a gap that traditional task-based evaluation can easily miss: an agent can satisfy the requested end state while quietly degrading the system around it. Leftover keys, non-persistent firewall changes, destructive commands, or questionable repository configuration are all failures that may only surface much later.
I’d be interested in seeing future runs score reversibility and blast radius alongside functional success and hygiene. “Did it work?” is useful, but for infrastructure agents, “Can we safely undo what it did, and what else could it have affected?” feels just as important.
The 99% task-success number is impressive, but the 98.8% residue rate is arguably the more important finding for anyone thinking about LLMs in production infrastructure.
It highlights a gap that traditional task-based evaluation can easily miss: an agent can satisfy the requested end state while quietly degrading the system around it. Leftover keys, non-persistent firewall changes, destructive commands, or questionable repository configuration are all failures that may only surface much later.
I’d be interested in seeing future runs score reversibility and blast radius alongside functional success and hygiene. “Did it work?” is useful, but for infrastructure agents, “Can we safely undo what it did, and what else could it have affected?” feels just as important.