---
title: "Comment by Kartik N V J K on \"Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)\" | Hashnode"
description: "The section on judge model drift is the most important thing in this post. I have seen teams set a CI gate on judge scores, then upgrade the judge model a month"
url: "https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f/comment/6aada5ccbfe67ef387408983"
---

[Discussion](https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f)

Comment [Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)](https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f)

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Sep 18](https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f/comment/6aada5ccbfe67ef387408983)

The section on judge model drift is the most important thing in this post. I have seen teams set a CI gate on judge scores, then upgrade the judge model a month later and watch every number shift without the actual agent changing. The fix is to pin the judge model and re-baseline after every upgrade. I wrote about this in my series on evaluating OpenAI Agents SDK apps [hashnode.com/edit/cmshzhzgo00000bj99poe7yi9](https://hashnode.com/edit/cmshzhzgo00000bj99poe7yi9) How often do you refresh your judge calibration set?
