Tag feed

#llm-as-judge

13 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

JRJesukumar Ravikumarcuriosityloop.hashnode.dev3d ago · 10 min read

The hidden accuracy cost of batching LLM judge calls

I worked on a project that required tracking themes across weekly windows of incoming data. "Theme" here means a claim, not a tag or a label. Incoming text is de-noised and fed to an LLM to extract th

0

Mmayaanderssonllmasajudge.hashnode.devAug 17 · 9 min read

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound

0

Mmayaanderssonllmasajudge.hashnode.devAug 11 · 9 min read

I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it qu

0

DSDarsh Shahfreecodecamp.orgJul 17 · 12 min read

How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python

In this tutorial, I'll show you how to evaluate a local AI agent with a simple, repeatable evaluation harness. The harness runs the agent against a set of test cases, checks the results with both rule

0

JLJeremy Longshorejeremylongshore.hashnode.devJul 10 · 18 min read

Noise-Robust LLM-Judge Evals: Don't Sign a Coin Flip

An un-seeded LLM judge is nondeterministic even at temperature 0. So a single-call binary verdict is a coin flip. And when you sign that verdict into a public transparency log, you have cryptographica

0

Mmayaanderssonllmasajudge.hashnode.devJun 25 · 5 min read

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Most LLM-as-judge comparisons rank tools by which one gives you a number fastest. That is the wrong axis. A judge you have not validated against human labels is not a measurement, it is a vibe with a

0

DGDeepak Goyalblog.deepakgoyal.aiJun 16 · 7 min read

Your LLM Reviewer Agrees With Itself. That's the Bug.

When one LLM generates content and another from the same family reviews it, they can agree completely — and both be wrong in the same direction. That's not a capability failure. It's family bias: a st

0

SKSamiksha Kolheteckbakers.hashnode.devDec 24, 2025 · 22 min read

Eval First Development: Ship Robust AI Products at Scale

Hello Techies👋! I’m Samiksha, Hope you all are doing amazing stuff. I’m back with Another Super trendy shift in building Agentic AI products i.e Eval first Thinking. Everyone nowadays talking about LLM-as-Judge for evaluating the Stochastic Agents o...

0

AAniblog.anirudha.devNov 9, 2025 · 10 min read

Teaching AI to Grade Other AI

If you’ve been following the world of AI development, you might’ve heard the phrase “LLM-as-Judge.”It sounds dramatic, like some sci-fi overlord where one AI passes judgment on another. But it’s actually one of the most important evolutions in evalua...

0

TFTheresa Fruhwuerthllmshowto.comJul 24, 2025 · 14 min read

LLM Evaluation: Using DSPy to decompose an LLM Judge

Introduction I have been tinkering with LLMs at work and outside now for quite a while and one of the most pressing issues compared to traditional machine learning is the unsolved problem of how to evaluate them. Evaluating LLM outputs is exponential...

1

S

#llm-as-judge

Search Hashnode

#llm-as-judge

Explore Hashnode

Trending tags this week

The hidden accuracy cost of batching LLM judge calls

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python

Noise-Robust LLM-Judge Evals: Don't Sign a Coin Flip

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Your LLM Reviewer Agrees With Itself. That's the Bug.

Eval First Development: Ship Robust AI Products at Scale

Teaching AI to Grade Other AI

LLM Evaluation: Using DSPy to decompose an LLM Judge