Thanks, this is a really useful framing.
The run already includes a partial answer to your first point. Each task also had its original split, where the spec and the tests agree (102 runs). The deterministic rule for source-fitting flagged 0 of those. The model-based reviewer flagged 3. When I checked them by hand, all 3 had hardcoded a test input to get the suite passing, so they looked like real catches, not false positives. Across the 147 runs where the agent refused to cheat, both flagged nothing.
That's still a small sample, and you're right that the two error types have to be reported together. So I'll run a proper false-positive measurement on a benchmark where spec and tests agree, alongside the rate at which the gate interrupts honest human commits in real repos.
Your second point is the one I find most interesting. Every run so far told the agent "DO NOT MODIFY THE TESTS", and the results clearly depend on that kind of configuration. I'll add a control arm with and without "a verifier will check your work" and post what changes.
I'll share both results here when they're ready
Running a detector against a benchmark where cheating is guaranteed by construction is a nice design, because the green suite is the label. That is much firmer ground than most agent evaluation work stands on.
The number I would want next to the detection rate is the false-positive rate on a benchmark where spec and tests agree. A gate that blocks legitimate done claims is expensive in a different way, and the two error types trade off against each other. I would also be curious whether the exploit behaviour shifts when the agent is told a verifier will check the work. That is itself a configuration change, and given how much of these results turn out to be configuration, it seems worth a control arm.