Your AI Agent Evaluation Is Lying to You: Why 10 Test Runs Prove Nothing

I ran 10 games between two AI agents. Agent v3 went 5-5 against Agent v1. I reported "v3 ties v1, no measurable improvement, don't merge." That conclusion was wrong. Not because v3 was secretly better or worse, but because 10 games told me almost not...