Discussion on "Don't trust the agent's PASS: a locked-test bench you can build in an hour"

Sai Ram Kotha · 2026-10-09T02:18:27.513Z

Every coding-agent comparison ends with a number. Most of them can't be checked. Here is a small bench design where every number can be re-run by anyone, and what my first pilot did and did not show.

Locking the test file with a SHA-256 and disqualifying any run that touches the hash is the right instinct, because the moment the agent can edit the scorer, PASS means nothing. The n=3 honesty is good to see too, most people would have shipped a ranking. I ran into the other half of this, a suite that stayed green while the agent still broke in production, and wrote up the CI/CD I moved to: kartiknvjk.hashnode.dev/my-agent-passed-every-che… . Are you adding held-out hidden tests on top of the locked file?

For the next task I'd hold some tests back. Three clean runs on six bugs show the bench works, but visible tests won't separate models much, because a failing test also shows the agent what a passing answer looks like.

Keep giving the agent the six tests, then score against a second file it never sees: the same bugs, checked with different inputs. For an inventory module that could be a negative quantity, or an item that isn't stocked yet. A fix that special-cases the visible inputs passes the first file and fails the second. The gap between those two scores is the number I'd want from a cross-model comparison, and it costs you one more locked hash.

This is the missing half. The pilot only showed the bench is reproducible, not that it discriminates. A held-out file with different inputs, locked by its own hash, would turn "passes" into "generalizes", and the visible-vs-hidden gap is a better cross-model number than either score. I'll add it as the next version of the bench and say so in the post. Which inputs would you pick first for the inventory module?

Locking the test file protects the assertions, but the independent rerun should also prove that all six intended tests actually executed against the edited inventory module. A green process exit can still come from empty discovery or importing an installed copy instead of the fresh workspace.

I would record discovered test IDs, executed count and the resolved module path alongside the test hash. Make a zero-test run fail the bench, and retain the six per-test outcomes rather than only Yes in the pilot table. Those checks preserve your small reproducible setup while making its PASS evidence about the intended implementation rather than just the test runner's exit status.

Good catch, and I'll say plainly that the pilot doesn't prove this. A zero-test run or a stale installed copy would also exit green. I'll record discovered IDs, executed count and the resolved module path next to the hash, fail on zero tests, and keep all six per-test results in the table. Thank you, this makes the evidence about the edited code rather than the runner.

Search Hashnode

Don't trust the agent's PASS: a locked-test bench you can build in an hour

Responses(5)