Discussion on "My test suite was green. My detector was wrong."

Sravan Vidiyala · 2026-08-10T05:24:33.336Z

Correction, 11 August 2026. The figures below were wrong when first published. My registry harvester paginated over overlapping pages and returned the same servers repeatedly, so I counted 467 servers

The part about testing your own assumptions really hit home. A green suite can just mean you built a perfect loop between the code and your fixtures.

Using real corpus data feels like the better sanity check, especially for security tools where normal words can look suspicious.

Green suite, wrong detector is the exact trap I keep warning people about: the tests validate your assumptions, not reality. I wrote up how I generate test cases that actually catch this class of bug: kartiknvjk.hashnode.dev/how-i-generate-llm-test-c… . How did you eventually catch that the detector was off, a production signal or a manual check?

Manual. The tool wasn't released yet, so there was no production to get a signal from.

I harvested 13,474 real tool descriptions, ran the detector over them of which twenty servers failed. All twenty were wrong.

Nothing prompted it. Nothing looked broken. I went looking because I'd written both the tests and the rules, so the tests passing didn't tell me anything.

That's the part worth generalising I suppose, that this cateogry of bug has no symptom. You don't catch it by watching, you catch it by getting data you didn't make up.

I only half-fixed it, mind. My "12 of 13 attack shapes" uses a list I wrote myself, so it has the same problem. That one's still open.

Search Hashnode

My test suite was green. My detector was wrong.

Responses(3)