---
title: "Comment by Kartik N V J K on \"LLM-as-Judge Is Not a Score. It Is a Reasoning Contract.\" | Hashnode"
description: "Clause 4, letting a rule engine veto rather than average a subjective score, is the one I wish more people implemented, since a judge that can be out-voted by a"
url: "https://hashnode.com/posts/llm-as-judge-is-not-a-score-it-is-a-reasoning-contract/6ac51941755f8dfd2cdaa354/comment/6ac93cc491361fd11a73254e"
---

[Discussion](https://hashnode.com/posts/llm-as-judge-is-not-a-score-it-is-a-reasoning-contract/6ac51941755f8dfd2cdaa354)

Comment [LLM-as-Judge Is Not a Score. It Is a Reasoning Contract.](https://hashnode.com/posts/llm-as-judge-is-not-a-score-it-is-a-reasoning-contract/6ac51941755f8dfd2cdaa354)

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Oct 9](https://hashnode.com/posts/llm-as-judge-is-not-a-score-it-is-a-reasoning-contract/6ac51941755f8dfd2cdaa354/comment/6ac93cc491361fd11a73254e)

Clause 4, letting a rule engine veto rather than average a subjective score, is the one I wish more people implemented, since a judge that can be out-voted by an objective check stops the confident-but-wrong score from leaking through. The fails/borderline/clears bands over fake decimals is exactly right too. How big is your frozen regression corpus before you trust a judge change?

[Harrison Guo](https://hashnode.com/@harrisonguo)

[Oct 10](https://hashnode.com/posts/llm-as-judge-is-not-a-score-it-is-a-reasoning-contract/6ac51941755f8dfd2cdaa354/comment/6ac93cc491361fd11a73254e#reply-6ac9d1d0291a1071816d0213)

Thanks Kartik. You got Clause 4 exactly: the veto only works if nothing is allowed to average it away.

On the corpus, I'd be guessing if I gave you a number, and the post doesn't give one either. What I watch is what's in the set, not how big it is. It's hand-labeled and versioned, and I seed it from two places: every case the judge got wrong before, and the borderline band. Clear passes almost never teach you anything. A judge change passes when nothing flipped between fails and clears, and I can explain each borderline move. One fails-to-clears flip is a regression even on a small set, and a huge set made of easy passes proves nothing.

It also keeps growing. Each time the judge and a human disagree in production, that case goes into the set, so it grows where the judge was weakest.