Discussion on "My AI Said "Fixed" Three Times. It Was Wrong Three Times."

Seiji Nakaya · 2026-08-11T22:40:18.753Z

I nearly shipped three broken fixes in a row because an AI told me they were done and I wanted to believe it. What stopped me was not a better model — it was a second agent that was not allowed to hav

The design choice hiding in your setup deserves top billing: the reviewer had access to tests the author's environment could not run, and you call that "useful, not inconvenient." That asymmetry is what makes "do not use a second AI as another voter. Use it as an independent source of friction" actually work. Two agents with identical checkouts, tools and context converge on identical blind spots, and the vote is theater; the reviewer that caught your ten findings caught them by running things the author physically could not. Which suggests a stronger version of your protocol: engineer the asymmetry deliberately, different tool access, different test tiers, maybe a different model family, so agreement carries information. The finding that the hardest round ended with less code, deleting the semantic detector instead of patching it a fifth time, is also the most human-senior-engineer behavior I have seen attributed to a review loop.

Honestly, we didn't design that part. The two agents ended up in different environments because of what we were building, not because I had a review method in mind: for a while I treated the gaps as something to fix. You've put it better than I did in the post: if agreement is going to mean anything, the asymmetry has to be deliberate. I'll take that as the rule now. The round that ended with less code was the uncomfortable one. Neither agent had anything invested in the four earlier fixes. I did.

Fixed is a claim about a system you have not re-broken yet. The gate is a second agent that has to reproduce the failure on a clean checkout before any fix lands. Wanting to believe the model is the whole problem. I see it fails the merge gate. Less code that clears that bar still wins over three green checkmarks that never saw the original break again.

Fixed is a claim about a system you have not re-broken yet", I wish I'd written that line.

I didn't adopt the reproduce-first gate as a principle either. I added it after getting burned, because I needed something I couldn't argue our way past. The model being wrong three times was normal. Me accepting it three times is what needed the gate.

Search Hashnode

My AI Said "Fixed" Three Times. It Was Wrong Three Times.

Responses4