---
title: "Discussion on \"Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)\" | Hashnode"
description: "Discussion on \"Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)\". Build an LLM-as-judge harness for AI agents: rubric design, the bias traps (position, verbosity, drift), and the anchor set that keeps it honest."
url: "https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Evaluating Agents With an LLM-as-Judge Harness (Without Kidding Yourself About It)](https://virginiamwegahashnodedev.hashnode.dev/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it?utm_source=hashnode&utm_medium=feed)

Build an LLM-as-judge harness for AI agents: rubric design, the bias traps (position, verbosity, drift), and the anchor set that keeps it honest.

Jul 1 · 7 min read · virginiamwegahashnodedev.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#ai](https://hashnode.com/tag/ai)
-   [#testing](https://hashnode.com/tag/testing)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#system-design](https://hashnode.com/tag/system-design)

## Written by

[Virginia Mwega](https://hashnode.com/@veemwega)

@veemwega · Nairobi,Kenya

  
[View profile](https://hashnode.com/@veemwega)

AI Developer Experience | OpenAI SDKs, AI API Integrations & LLM Observability

I build and document AI API integrations, test failure paths, and explore LLM observability and evaluation. I write about what breaks when AI meets real applications, how I investigate it, and what makes integrations easier for developers to use and trust.

## Responses 2

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Sep 18](https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f/comment/6aada5ccbfe67ef387408983)

The section on judge model drift is the most important thing in this post. I have seen teams set a CI gate on judge scores, then upgrade the judge model a month later and watch every number shift without the actual agent changing. The fix is to pin the judge model and re-baseline after every upgrade. I wrote about this in my series on evaluating OpenAI Agents SDK apps [hashnode.com/edit/cmshzhzgo00000bj99poe7yi9](https://hashnode.com/edit/cmshzhzgo00000bj99poe7yi9) How often do you refresh your judge calibration set?

[Ken](https://hashnode.com/@kenerator)

[Jul 7](https://hashnode.com/posts/evaluating-agents-with-an-llm-as-judge-harness-without-kidding-yourself-about-it/6a4532d8c925222f201f936f/comment/6a4c5fa85582aa8008f7e8db)

Strong write-up. The anchor-set point is the part I’d underline: once the judge is also a model, the harness needs its own calibration surface, not just a better judging prompt.

One enhancement I’d consider is making the eval receipt explicit for every run: judge version, candidate model, rubric version, anchor-set agreement, order-shuffle result, drift signal, and which failures were deterministic versus judgment-based. That makes the dashboard more inspectable when a score changes.

Affiliation note: I’m with nxus.SYSTEMS. This overlaps with nxusKit SDK CE examples around model-research harnesses, structured output, deterministic checks, Bayesian confidence, and retry/fallback patterns. CE is always free.

The easiest way to find the examples is to search for: nxusKit SDK examples
