---
title: "Discussion on \"My test suite was green. My detector was wrong.\" | Hashnode"
description: "Discussion on \"My test suite was green. My detector was wrong.\". I built a scanner to catch poisoned MCP tool descriptions, pointed it at 13,474 real ones, and watched it flag a tool called prompt_injection_scan as prompt injection."
url: "https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [My test suite was green. My detector was wrong.](https://blog.sravanvidiyala.com/my-test-suite-was-green-my-detector-was-wrong?utm_source=hashnode&utm_medium=feed)

I built a scanner to catch poisoned MCP tool descriptions, pointed it at 13,474 real ones, and watched it flag a tool called prompt\_injection\_scan as prompt injection.

Aug 10 · 9 min read · blog.sravanvidiyala.com  
Updated Aug 12  
Read the article

-   [#ai-agents](https://hashnode.com/tag/ai-agents)
-   [#security](https://hashnode.com/tag/security)
-   [#testing](https://hashnode.com/tag/testing)
-   [#python](https://hashnode.com/tag/python)
-   [#mcp](https://hashnode.com/tag/mcp)
-   [#promptinjections](https://hashnode.com/tag/promptinjections)

## Written by

[Sravan Vidiyala](https://hashnode.com/@sravanvidiyala)

@sravanvidiyala · Sydney, Australia

  
[View profile](https://hashnode.com/@sravanvidiyala)

Engineering LLM systems that hold in production.

I build LLM-powered systems and write about the engineering that keeps them safe in production — scoping, deterministic verification, and governance.

## Responses 3

Markdown supported · ⌘↵ to post

[Julian Neagu](https://hashnode.com/@julianneagulab)

[Aug 15](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008/comment/6a8051e4bda9b0232e687b98)

The part about testing your own assumptions really hit home. A green suite can just mean you built a perfect loop between the code and your fixtures.

Using real corpus data feels like the better sanity check, especially for security tools where normal words can look suspicious.

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Aug 10](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008/comment/6a7a0cfcdd81a7d4712b5533)

Green suite, wrong detector is the exact trap I keep warning people about: the tests validate your assumptions, not reality. I wrote up how I generate test cases that actually catch this class of bug: [kartiknvjk.hashnode.dev/how-i-generate-llm-test-c…](https://kartiknvjk.hashnode.dev/how-i-generate-llm-test-cases-that-actually-catch-bugs) . How did you eventually catch that the detector was off, a production signal or a manual check?

[Sravan Vidiyala](https://hashnode.com/@sravanvidiyala)

[Aug 10](https://hashnode.com/posts/my-test-suite-was-green-my-detector-was-wrong/6a796091a1f0931c50030008/comment/6a7a0cfcdd81a7d4712b5533#reply-6a7a5cfd0505268b37c6be83)

Manual. The tool wasn't released yet, so there was no production to get a signal from.

I harvested 13,474 real tool descriptions, ran the detector over them of which twenty servers failed. All twenty were wrong.

Nothing prompted it. Nothing looked broken. I went looking because I'd written both the tests and the rules, so the tests passing didn't tell me anything.

That's the part worth generalising I suppose, that this cateogry of bug has no symptom. You don't catch it by watching, you catch it by getting data you didn't make up.

I only half-fixed it, mind. My "12 of 13 attack shapes" uses a list I wrote myself, so it has the same problem. That one's still open.
