---
title: "#llm-as-judge articles | Hashnode"
description: "Explore 13 articles tagged #llm-as-judge on Hashnode. Tutorials, guides, and how-tos from the community."
url: "https://hashnode.com/tag/llm-as-judge"
---

Tag

# #llm-as-judge

13 posts · 0 followers

[Jesukumar Ravikumar](https://hashnode.com/@jesukumar) · [Aug 26](https://hashnode.com/posts/the-hidden-accuracy-cost-of-batching-llm-judge-calls/6a8f77ef4df02f6c9638b0ec)

### [The hidden accuracy cost of batching LLM judge calls](https://curiosityloop.hashnode.dev/the-hidden-accuracy-cost-of-batching-llm-judge-calls?utm_source=hashnode&utm_medium=feed)

I worked on a project that required tracking themes across weekly windows of incoming data. "Theme" here means a claim, not a tag or a label. Incoming text is de-noised and fed to an LLM to extract th

10 min read · [curiosityloop.hashnode.dev](https://curiosityloop.hashnode.dev/the-hidden-accuracy-cost-of-batching-llm-judge-calls?utm_source=hashnode&utm_medium=feed)

[mayaandersson](https://hashnode.com/@mayaanderssondev) · [Aug 17](https://hashnode.com/posts/a-judge-that-agrees-with-your-humans-92-percent-of-the-time-can-be-at-60-percent-where-the-gate-actually-decides/6a83477a7b7f90bd9fc771c4)

### [A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides](https://llmasajudge.hashnode.dev/a-judge-that-agrees-with-your-humans-92-percent-of-the-time-can-be-at-60-percent-where-the-gate-actually-decides?utm_source=hashnode&utm_medium=feed)

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound

9 min read · [llmasajudge.hashnode.dev](https://llmasajudge.hashnode.dev/a-judge-that-agrees-with-your-humans-92-percent-of-the-time-can-be-at-60-percent-where-the-gate-actually-decides?utm_source=hashnode&utm_medium=feed)

[mayaandersson](https://hashnode.com/@mayaanderssondev) · [Aug 11](https://hashnode.com/posts/i-read-the-metric-libraries-of-five-widely-used-eval-tools-the-metric-was-never-the-hard-part/6a7b6feada84d66b0fbb7f1b)

### [I read the metric libraries of five widely-used eval tools. The metric was never the hard part.](https://llmasajudge.hashnode.dev/i-read-the-metric-libraries-of-five-widely-used-eval-tools-the-metric-was-never-the-hard-part?utm_source=hashnode&utm_medium=feed)

Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it qu

9 min read · [llmasajudge.hashnode.dev](https://llmasajudge.hashnode.dev/i-read-the-metric-libraries-of-five-widely-used-eval-tools-the-metric-was-never-the-hard-part?utm_source=hashnode&utm_medium=feed)

[Darsh Shah](https://hashnode.com/@darshs) · [Jul 17](https://hashnode.com/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python/6a5a98bcef0967f8fb858895)

### [How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python](https://freecodecamp.org/news/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python?utm_source=hashnode&utm_medium=feed)

In this tutorial, I'll show you how to evaluate a local AI agent with a simple, repeatable evaluation harness. The harness runs the agent against a set of test cases, checks the results with both rule

12 min read · [freecodecamp.org](https://freecodecamp.org/news/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python?utm_source=hashnode&utm_medium=feed)

[Jeremy Longshore](https://hashnode.com/@jeremylongshore) · [Jul 10](https://hashnode.com/posts/noise-robust-llm-judge-evals-dont-sign-a-coin-flip/6a50c95adcb51279bdac17d0)

### [Noise-Robust LLM-Judge Evals: Don't Sign a Coin Flip](https://jeremylongshore.hashnode.dev/noise-robust-llm-judge-evals-dont-sign-a-coin-flip?utm_source=hashnode&utm_medium=feed)

An un-seeded LLM judge is nondeterministic even at temperature 0. So a single-call binary verdict is a coin flip. And when you sign that verdict into a public transparency log, you have cryptographica

18 min read · [jeremylongshore.hashnode.dev](https://jeremylongshore.hashnode.dev/noise-robust-llm-judge-evals-dont-sign-a-coin-flip?utm_source=hashnode&utm_medium=feed)

[mayaandersson](https://hashnode.com/@mayaanderssondev) · [Jun 25](https://hashnode.com/posts/i-checked-six-llm-as-judge-tools-against-human-labels-the-scoreboard-was-the-wrong-thing-to-read/6a3d6b63027167332e908296)

### [I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.](https://llmasajudge.hashnode.dev/i-checked-six-llm-as-judge-tools-against-human-labels-the-scoreboard-was-the-wrong-thing-to-read?utm_source=hashnode&utm_medium=feed)

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

5 min read · [llmasajudge.hashnode.dev](https://llmasajudge.hashnode.dev/i-checked-six-llm-as-judge-tools-against-human-labels-the-scoreboard-was-the-wrong-thing-to-read?utm_source=hashnode&utm_medium=feed)

[Deepak Goyal](https://hashnode.com/@deepakgoyal-ai) · [Jun 16](https://hashnode.com/posts/llm-reviewer-agrees-with-itself-thats-the-bug/6a313c94e260327300316650)

### [Your LLM Reviewer Agrees With Itself. That's the Bug.](https://blog.deepakgoyal.ai/llm-reviewer-agrees-with-itself-thats-the-bug?utm_source=hashnode&utm_medium=feed)

Single-LLM review shares blind spots with the model it's reviewing. Here's the two-judge convergence rule I built, and the open-source tool to run it yourself.

7 min read · [blog.deepakgoyal.ai](https://blog.deepakgoyal.ai/llm-reviewer-agrees-with-itself-thats-the-bug?utm_source=hashnode&utm_medium=feed)

[Samiksha Kolhe](https://hashnode.com/@AInaut) · [Dec 24, 2025](https://hashnode.com/posts/eval-first-approach/694be26869ca7b341b89aa95)

### [Eval First Development: Ship Robust AI Products at Scale](https://teckbakers.hashnode.dev/eval-first-approach?utm_source=hashnode&utm_medium=feed)

Hello Techies👋! I’m Samiksha, Hope you all are doing amazing stuff. I’m back with Another Super trendy shift in building Agentic AI products i.e Eval first Thinking. Everyone nowadays talking about LLM-as-Judge for evaluating the Stochastic Agents o...

22 min read · [teckbakers.hashnode.dev](https://teckbakers.hashnode.dev/eval-first-approach?utm_source=hashnode&utm_medium=feed)

[Ani](https://hashnode.com/@anirudha) · [Nov 9, 2025](https://hashnode.com/posts/llm-as-judge/691055dfd1edb784ad7b84c6)

### [Teaching AI to Grade Other AI](https://blog.anirudha.dev/llm-as-judge?utm_source=hashnode&utm_medium=feed)

Understanding LLM as Judge

10 min read · [blog.anirudha.dev](https://blog.anirudha.dev/llm-as-judge?utm_source=hashnode&utm_medium=feed)

[Theresa Fruhwuerth](https://hashnode.com/@TheresaFruhwuerth) · [Jul 24, 2025](https://hashnode.com/posts/llm-evaluation-using-dspy-to-decompose-an-llm-judge/688230c1a8193e636755d655)

### [LLM Evaluation: Using DSPy to decompose an LLM Judge](https://llmshowto.com/llm-evaluation-using-dspy-to-decompose-an-llm-judge?utm_source=hashnode&utm_medium=feed)

Key Reasons to Use DSPy for LLM Judge Decomposition

14 min read · [llmshowto.com](https://llmshowto.com/llm-evaluation-using-dspy-to-decompose-an-llm-judge?utm_source=hashnode&utm_medium=feed)

[1](https://hashnode.com/posts/llm-evaluation-using-dspy-to-decompose-an-llm-judge/688230c1a8193e636755d655)
