Tag feed

#llm-evaluation

32 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

MAMuhammad Azlaan Zubairblog.mdazlaanzubair.com1d ago · 6 min read

Benchmark the Model–Harness Pair, Not the Model

“Which model is best for coding?” sounds like a precise engineering question. Usually, it is missing half of the system. A coding agent is a model inside a harness that chooses tools, assembles contex

2

R

Mmayaanderssonllmasajudge.hashnode.devAug 17 · 9 min read

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound

0

Mmarcuschenvoicelatency.hashnode.devAug 13 · 13 min read

The call failed on turn nine. My eval gave me one number for the whole call.

The transcript was fourteen turns long and the score was 0.62. That is the entire output. One float, one call, and a rubric that said something like "did the agent resolve the customer's issue." It di

0

Mmayaanderssonllmasajudge.hashnode.devAug 12 · 5 min read

Run forty experiments against one eval set and you will find an improvement that is not there

You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an

0

Mmayaanderssonllmasajudge.hashnode.devAug 11 · 9 min read

I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it qu

0

RRRedwan Rahmannirizan.hashnode.devAug 10 · 8 min read

The Ruler Can Change Too: Navigating Judge Drift in Production AI Evaluation

The Monday Morning Regression It is Monday morning. You arrive at your desk to find that production monitoring has flagged a sudden, severe degradation in your core Retrieval-Augmented Generation (RAG

2

K

Mmayaanderssonllmasajudge.hashnode.devAug 7 · 10 min read

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate

0

Mmayaanderssonllmasajudge.hashnode.devAug 6 · 12 min read

Upgrading the judge ends one score series and starts another

There is a mature literature on what happens when you swap one measuring instrument for another, and it is not in machine learning. The standard treatment is Bland and Altman, "Statistical methods for

0

Mmayaanderssonllmasajudge.hashnode.devAug 5 · 15 min read

A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.

Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All

0

MSManu Shuklaecorpit.hashnode.devAug 1 · 14 min read

LLM tool-use reliability in 2026: how to evaluate models for AI agents

LLM tool-use reliability in 2026: how to evaluate models for AI agents Summary. The benchmark that predicts whether an AI agent survives production is not the one most teams check. The Berkeley τ-benc

0

#llm-evaluation

Search Hashnode

#llm-evaluation

Explore Hashnode

Trending tags this week

Benchmark the Model–Harness Pair, Not the Model

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

The call failed on turn nine. My eval gave me one number for the whole call.

Run forty experiments against one eval set and you will find an improvement that is not there

I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

The Ruler Can Change Too: Navigating Judge Drift in Production AI Evaluation

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

Upgrading the judge ends one score series and starts another

A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.

LLM tool-use reliability in 2026: how to evaluate models for AI agents