---
title: "Comment by Mike Dabydeen on \"Don't trust the agent's PASS: a locked-test bench you can build in an hour\" | Hashnode"
description: "For the next task I'd hold some tests back. Three clean runs on six bugs show the bench works, but visible tests won't separate models much, because a failing t"
url: "https://hashnode.com/posts/dont-trust-agent-pass-locked-test-bench/6ac84ef3a3016099157b4006/comment/6ac8cd93cdb37db53d67795c"
---

[Discussion](https://hashnode.com/posts/dont-trust-agent-pass-locked-test-bench/6ac84ef3a3016099157b4006)

Comment [Don't trust the agent's PASS: a locked-test bench you can build in an hour](https://hashnode.com/posts/dont-trust-agent-pass-locked-test-bench/6ac84ef3a3016099157b4006)

[Mike Dabydeen](https://hashnode.com/@firelinks)

[Oct 9](https://hashnode.com/posts/dont-trust-agent-pass-locked-test-bench/6ac84ef3a3016099157b4006/comment/6ac8cd93cdb37db53d67795c)

For the next task I'd hold some tests back. Three clean runs on six bugs show the bench works, but visible tests won't separate models much, because a failing test also shows the agent what a passing answer looks like.

Keep giving the agent the six tests, then score against a second file it never sees: the same bugs, checked with different inputs. For an inventory module that could be a negative quantity, or an item that isn't stocked yet. A fix that special-cases the visible inputs passes the first file and fails the second. The gap between those two scores is the number I'd want from a cross-model comparison, and it costs you one more locked hash.

[Sai Ram Kotha](https://hashnode.com/@sairamkotha)

[Oct 9](https://hashnode.com/posts/dont-trust-agent-pass-locked-test-bench/6ac84ef3a3016099157b4006/comment/6ac8cd93cdb37db53d67795c#reply-6ac8f926579ef132773cd0af)

This is the missing half. The pilot only showed the bench is reproducible, not that it discriminates. A held-out file with different inputs, locked by its own hash, would turn "passes" into "generalizes", and the visible-vs-hidden gap is a better cross-model number than either score. I'll add it as the next version of the bench and say so in the post. Which inputs would you pick first for the inventory module?

[Mike Dabydeen](https://hashnode.com/@firelinks)

[Oct 12](https://hashnode.com/posts/dont-trust-agent-pass-locked-test-bench/6ac84ef3a3016099157b4006/comment/6ac8cd93cdb37db53d67795c#reply-6accc14805ad79716a6c5972)

I'd start with the cases right next to each visible test, because that is where a special-cased fix breaks first. Two I'd pick for an inventory module: removing exactly the quantity left in stock, which catches a < that should be <=, and the same SKU appearing twice in one order, which catches code that checks each line against the starting stock instead of what is left after the first line.

Your write-the-tests-first step carries over too: run the hidden file against the buggy module and confirm every hidden test fails. A hidden test that passes on broken code passes for every model, so the hidden score looks better than it is and the gap shrinks.
