---
title: "Discussion on \"A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.\" | Hashnode"
description: "Discussion on \"A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.\". Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All"
url: "https://hashnode.com/posts/a-noisy-judge-does-not-just-add-error-bars-it-shrinks-the-effect-you-are-trying-to-measure/6a738ec23b6378c3ed95dc7d"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.](https://llmasajudge.hashnode.dev/a-noisy-judge-does-not-just-add-error-bars-it-shrinks-the-effect-you-are-trying-to-measure?utm_source=hashnode&utm_medium=feed)

Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All

Aug 5 · 15 min read · llmasajudge.hashnode.dev  
Read the article

-   [#llm-evaluation](https://hashnode.com/tag/llm-evaluation)
-   [#statistics](https://hashnode.com/tag/statistics)
-   [#datascience](https://hashnode.com/tag/datascience)
-   [#testing](https://hashnode.com/tag/testing)

## Written by

[mayaandersson](https://hashnode.com/@mayaanderssondev)

@mayaanderssondev · Palo Alto CA

  
[View profile](https://hashnode.com/@mayaanderssondev)

Just a bored curious dev

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
