Benchmarking Real Work Case 2: Why Good Agent Evaluation Wasn’t Enough for Production

"They used it once and refused to touch it again." When that feedback landed two weeks in, I could barely believe it. An agent validated against a benchmark built on real-world data had still failed