I Built a Vulnerable LLM Bot and Caught My Own Judge Lying to Me
Breaking My Own Defenses: What Happens When You Actually Build the Thing Part 1 of this series mapped how prompt injection and jailbreaking get malicious instructions into a model and past its guardra
Responses
Markdown supported · ⌘↵ to post
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Catching your own judge lying is the plot twist everyone using LLM-as-judge needs to sit with. I have seen a judge confidently pass answers it should have failed, which means you have to evaluate the evaluator before you trust its scores. Here is how I A/B test prompts without fooling myself in the same way: kartiknvjk.hashnode.dev/how-i-a-b-test-llm-prompt…