test_counts_at_runtime is a nasty one. An ImportError dressed as a red test reads exactly like a real catch in a CI summary. I do a version of Check 1 for eval cases: a new case has to fail on the old prompt before I believe it guards anything. To your question, I'd run Check 1 on every agent PR and save hand-written mutants for diffs that touch a boundary like that range() bound.
That's the same move, and I like it for evals: a new case that passes on the old prompt guards nothing. Your split matches mine too: Check 1 everywhere, hand mutants where the diff touches a boundary.
How do you handle nondeterminism there? Does a new eval case have to fail on the old prompt every run, or most of them?
Same filter as Check 1: the red only narrows it down, the failure line decides. If the test fails on an assertion about the bug's behavior, it counts. If it fails because the revert took away something it needs, like an import, a helper or a fixture (the test_counts_at_runtime case), it's noise, not evidence.
When a whole-file revert breaks things that aren't the bug, revert less: undo only the hunk that fixed it, which is really the hand-written mutant from Check 2. And if the test can't even run without the new code, that tells you something too. It's tied to the new API rather than to the behavior, so I'd rewrite it against behavior that exists on both sides before trusting it.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
test_counts_at_runtime is a nasty one. An ImportError dressed as a red test reads exactly like a real catch in a CI summary. I do a version of Check 1 for eval cases: a new case has to fail on the old prompt before I believe it guards anything. To your question, I'd run Check 1 on every agent PR and save hand-written mutants for diffs that touch a boundary like that range() bound.