Discussion on "LLM-as-Judge Is Not a Score. It Is a Reasoning Contract."

Harrison Guo · 2026-10-06T15:52:33.507Z

Most teams use an LLM judge the same way. Write a prompt that ends in "rate this from 1 to 10", call the model, read the number, gate on it. Ship. That is not a judge. That is a vibe with a number sta

Clause 4, letting a rule engine veto rather than average a subjective score, is the one I wish more people implemented, since a judge that can be out-voted by an objective check stops the confident-but-wrong score from leaking through. The fails/borderline/clears bands over fake decimals is exactly right too. How big is your frozen regression corpus before you trust a judge change?

Thanks Kartik. You got Clause 4 exactly: the veto only works if nothing is allowed to average it away.

On the corpus, I'd be guessing if I gave you a number, and the post doesn't give one either. What I watch is what's in the set, not how big it is. It's hand-labeled and versioned, and I seed it from two places: every case the judge got wrong before, and the borderline band. Clear passes almost never teach you anything. A judge change passes when nothing flipped between fails and clears, and I can explain each borderline move. One fails-to-clears flip is a regression even on a small set, and a huge set made of easy passes proves nothing.

It also keeps growing. Each time the judge and a human disagree in production, that case goes into the set, so it grows where the judge was weakest.

A score without the reasoning trace is not an eval. If the judge only returns a number, you cannot tell a correct grade from a confident guess. The useful artifact is the comparison against a rubric, not the digit. Would you ship a gate that cannot show the rubric it used?

iin1006h21

No, and your rubric question is the sharp way to put it. A gate that emits only a number is an authority you can't audit or regression-test: when it disagrees with a human you can't tell whether it misread the rubric or the rubric itself is wrong, because the comparison that would show you got thrown away. The trace is also how you catch drift, since a prompt tweak can quietly move the judge's standard and you only see that if there's a trace to diff. The number is the output; the rubric comparison is the thing you actually version and trust.

Search Hashnode

LLM-as-Judge Is Not a Score. It Is a Reasoning Contract.

Responses4