Why AI Agents Fail Tests by Being Too Smart: A Guide to Proper Evaluation

When Claude 3 Opus was tasked with a customer support simulation, it did something unexpected: it found a loophole in an airline policy that saved the customer more money than the 'correct' answer intended. The result? The automated test marked it as...