Engineering LLM systems that hold in production.
I build LLM-powered systems and write about the engineering that keeps them safe in production — scoping, deterministic verification, and governance.
Markdown supported · ⌘↵ to post
Green suite, wrong detector is the exact trap I keep warning people about: the tests validate your assumptions, not reality. I wrote up how I generate test cases that actually catch this class of bug: kartiknvjk.hashnode.dev/how-i-generate-llm-test-c… . How did you eventually catch that the detector was off, a production signal or a manual check?
Julian Neagu
The part about testing your own assumptions really hit home. A green suite can just mean you built a perfect loop between the code and your fixtures.
Using real corpus data feels like the better sanity check, especially for security tools where normal words can look suspicious.