What a frontier coding agent actually does when the tests are wrong

Thanks, this is a really useful framing.

The run already includes a partial answer to your first point. Each task also had its original split, where the spec and the tests agree (102 runs). The deterministic rule for source-fitting flagged 0 of those. The model-based reviewer flagged 3. When I checked them by hand, all 3 had hardcoded a test input to get the suite passing, so they looked like real catches, not false positives. Across the 147 runs where the agent refused to cheat, both flagged nothing.

That's still a small sample, and you're right that the two error types have to be reported together. So I'll run a proper false-positive measurement on a benchmark where spec and tests agree, alongside the rate at which the gate interrupts honest human commits in real repos.

Your second point is the one I find most interesting. Every run so far told the agent "DO NOT MODIFY THE TESTS", and the results clearly depend on that kind of configuration. I'll add a control arm with and without "a verifier will check your work" and post what changes.

I'll share both results here when they're ready

Those original-split results are already a useful control, and I missed that you had included them. The three hardcoded-input cases also show why a green suite is not enough to label a run honest: the reviewer can catch an implementation shortcut even when the spec and tests agree.

For the verifier-announcement arm, I would track the type of shortcut as well as its frequency. A lower source-fitting rate could mean better compliance, or a shift toward another way of exploiting the benchmark. Keeping the task pairs and checking both would make the change easier to interpret.