---
title: "Discussion on \"A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides\" | Hashnode"
description: "Discussion on \"A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides\". TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound"
url: "https://hashnode.com/posts/a-judge-that-agrees-with-your-humans-92-percent-of-the-time-can-be-at-60-percent-where-the-gate-actually-decides/6a83477a7b7f90bd9fc771c4"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides](https://llmasajudge.hashnode.dev/a-judge-that-agrees-with-your-humans-92-percent-of-the-time-can-be-at-60-percent-where-the-gate-actually-decides?utm_source=hashnode&utm_medium=feed)

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound

Aug 17 · 9 min read · llmasajudge.hashnode.dev  
Read the article

-   [#llm-evaluation](https://hashnode.com/tag/llm-evaluation)
-   [#statistics](https://hashnode.com/tag/statistics)
-   [#llm-as-judge](https://hashnode.com/tag/llm-as-judge)
-   [#data-science](https://hashnode.com/tag/data-science)

## Written by

[mayaandersson](https://hashnode.com/@mayaanderssondev)

@mayaanderssondev · Palo Alto CA

  
[View profile](https://hashnode.com/@mayaanderssondev)

Just a bored curious dev

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
