---
title: "Comment by Kartik N V J K on \"My test suite was green. My detector was wrong.\" | Hashnode"
description: "Green suite, wrong detector is the exact trap I keep warning people about: the tests validate your assumptions, not reality. I wrote up how I generate test case"
url: "https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008/comment/6a7a0cfcdd81a7d4712b5533"
---

[Discussion](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008)

Comment [My test suite was green. My detector was wrong.](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008)

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Aug 10](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008/comment/6a7a0cfcdd81a7d4712b5533)

Green suite, wrong detector is the exact trap I keep warning people about: the tests validate your assumptions, not reality. I wrote up how I generate test cases that actually catch this class of bug: [kartiknvjk.hashnode.dev/how-i-generate-llm-test-c…](https://kartiknvjk.hashnode.dev/how-i-generate-llm-test-cases-that-actually-catch-bugs) . How did you eventually catch that the detector was off, a production signal or a manual check?

[Sravan Vidiyala](https://hashnode.com/@sravanvidiyala)

[Aug 10](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008/comment/6a7a0cfcdd81a7d4712b5533#reply-6a7a5cfd0505268b37c6be83)

Manual. The tool wasn't released yet, so there was no production to get a signal from.

I harvested 13,474 real tool descriptions, ran the detector over them of which twenty servers failed. All twenty were wrong.

Nothing prompted it. Nothing looked broken. I went looking because I'd written both the tests and the rules, so the tests passing didn't tell me anything.

That's the part worth generalising I suppose, that this cateogry of bug has no symptom. You don't catch it by watching, you catch it by getting data you didn't make up.

I only half-fixed it, mind. My "12 of 13 attack shapes" uses a list I wrote myself, so it has the same problem. That one's still open.
