---
title: "Comment by Kartik N V J K on \"Evals: How Do You Actually Know Your AI Feature Works?\" | Hashnode"
description: "The pitfall you flag, eval scores climbing while user complaints stay flat, is the one that bit me hardest. A judge-model update changed its output format, my p"
url: "https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac6962204eafa6d98700716"
---

[Discussion](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c)

Comment [Evals: How Do You Actually Know Your AI Feature Works?](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c)

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Oct 7](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac6962204eafa6d98700716)

The pitfall you flag, eval scores climbing while user complaints stay flat, is the one that bit me hardest. A judge-model update changed its output format, my parser quietly fell back to a middle score, and the dashboard drifted up while nothing had improved. I wrote about this here: [kartiknvjk.hashnode.dev/i-built-an-llm-eval-frame…](https://kartiknvjk.hashnode.dev/i-built-an-llm-eval-framework-from-scratch-here-is-what-i-wish-i-had-bought-instead) Do you re-validate your judge against human labels on a set cadence, or only when a number looks wrong?

[Nagappan S](https://hashnode.com/@imkarthik03)

[Oct 13](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac6962204eafa6d98700716#reply-6acdd311d57024c379df180c)

That silent fallback is a nasty one. A parse failure that returns a middle score looks like a result. I'd make the parser fail loudly and track the parse-failure rate as its own metric.

For the judge, I'd do both. Re-check it against a small human-labelled set on a fixed cadence, and also whenever the judge model or its prompt changes. Pinning the judge version helps, because then drift only happens when you choose it. A sudden shift in the score distribution with no system change is a good trigger too.
