Discussion on "Don't trust the agent's PASS: a locked-test bench you can build in an hour"

Sai Ram Kotha · 2026-10-09T02:18:27.513Z

Every coding-agent comparison ends with a number. Most of them can't be checked. Here is a small bench design where every number can be re-run by anyone, and what my first pilot did and did not show.

Locking the test file with a SHA-256 and disqualifying any run that touches the hash is the right instinct, because the moment the agent can edit the scorer, PASS means nothing. The n=3 honesty is good to see too, most people would have shipped a ranking. I ran into the other half of this, a suite that stayed green while the agent still broke in production, and wrote up the CI/CD I moved to: kartiknvjk.hashnode.dev/my-agent-passed-every-che… . Are you adding held-out hidden tests on top of the locked file?

For the next task I'd hold some tests back. Three clean runs on six bugs show the bench works, but visible tests won't separate models much, because a failing test also shows the agent what a passing answer looks like.

Keep giving the agent the six tests, then score against a second file it never sees: the same bugs, checked with different inputs. For an inventory module that could be a negative quantity, or an item that isn't stocked yet. A fix that special-cases the visible inputs passes the first file and fails the second. The gap between those two scores is the number I'd want from a cross-model comparison, and it costs you one more locked hash.

This is the missing half. The pilot only showed the bench is reproducible, not that it discriminates. A held-out file with different inputs, locked by its own hash, would turn "passes" into "generalizes", and the visible-vs-hidden gap is a better cross-model number than either score. I'll add it as the next version of the bench and say so in the post. Which inputs would you pick first for the inventory module?

I'd start with the cases right next to each visible test, because that is where a special-cased fix breaks first. Two I'd pick for an inventory module: removing exactly the quantity left in stock, which catches a < that should be <=, and the same SKU appearing twice in one order, which catches code that checks each line against the starting stock instead of what is left after the first line.

Your write-the-tests-first step carries over too: run the hidden file against the buggy module and confirm every hidden test fails. A hidden test that passes on broken code passes for every model, so the hidden score looks better than it is and the gap shrinks.

Locking the test file protects the assertions, but the independent rerun should also prove that all six intended tests actually executed against the edited inventory module. A green process exit can still come from empty discovery or importing an installed copy instead of the fresh workspace.

I would record discovered test IDs, executed count and the resolved module path alongside the test hash. Make a zero-test run fail the bench, and retain the six per-test outcomes rather than only Yes in the pilot table. Those checks preserve your small reproducible setup while making its PASS evidence about the intended implementation rather than just the test runner's exit status.

Good catch, and I'll say plainly that the pilot doesn't prove this. A zero-test run or a stale installed copy would also exit green. I'll record discovered IDs, executed count and the resolved module path next to the hash, fail on zero tests, and keep all six per-test results in the table. Thank you, this makes the evidence about the edited code rather than the runner.

Recording all six outcomes will make the revised table much easier to audit. I would count skipped and expected-failure cases separately from executed passes, so collecting six tests cannot accidentally be reported as six checks of the implementation.

One cheap positive control is to run the bench against the original known-broken module and require the intended failures before accepting any repaired run. That verifies the locked assertions can still detect the six bugs with the current runner and import configuration.

Search Hashnode

Don't trust the agent's PASS: a locked-test bench you can build in an hour

Responses7