Your LLM Is Lying to You and You Have No Idea

The moment that changed how I think about this happened on a Tuesday afternoon. I was looking at our support assistant metrics. Latency was good. Error rate was basically zero. Token usage was normal.