Fantastic article and research, Mahdi! The observation that "green checkmarks mean less than they used to" because the same model writes both the code and the tests from its own biased picture is one of the most important truths in agentic engineering right now.
The experimental results on claimdesk (especially comparing M's predictive hallucinations against empirical explorers) clearly prove that there is no substitute for actually touching the running territory.
We work heavily on contract-driven agent governance, and we see your exploratory-testing-skill as the ideal companion to formal contracts: using chartered exploration to actively hunt for "green defects" across multi-step state, and then funneling those findings into strict, immutable regression contracts so the system compounds institutional memory over time.
Thank you for sharing the charters and the benchmark data—really valuable contribution to the agent engineering community!
Menno Drescher
Fantastic article and research, Mahdi! The observation that "green checkmarks mean less than they used to" because the same model writes both the code and the tests from its own biased picture is one of the most important truths in agentic engineering right now.
The experimental results on claimdesk (especially comparing M's predictive hallucinations against empirical explorers) clearly prove that there is no substitute for actually touching the running territory.
We work heavily on contract-driven agent governance, and we see your exploratory-testing-skill as the ideal companion to formal contracts: using chartered exploration to actively hunt for "green defects" across multi-step state, and then funneling those findings into strict, immutable regression contracts so the system compounds institutional memory over time.
Thank you for sharing the charters and the benchmark data—really valuable contribution to the agent engineering community!