You can't prove an agent works. Design it so you know when it doesn't.
Here's a moment most people hit when using agents to write code: it writes a test, the test passes, and you feel good about it. Then you run the real thing, and the code does something completely diff