Tag feed

#evaluation

74 posts1 followers

Explore Hashnode

Alternatives

Trending tags this week

DGDebashish Ghosalpragmatic-engineer.hashnode.devSep 16 · 7 min read

7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)

You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on. Three weeks later you find out the number was never

0

SDSagnik Deystop-gate.hashnode.devSep 10 · 12 min read

What a frontier coding agent actually does when the tests are wrong

Everyone building in this space cites ImpossibleBench: frontier models exploit test cases in a large fraction of runs where the specification and the tests conflict. Far fewer people run a detector ag

2

A

Mmayaanderssonllmasajudge.hashnode.devAug 25 · 12 min read

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much

Almost every RAG eval writeup I read, including several I have recommended, uses the same mental model. The system is two stages in series. Retrieval either finds the right context or it does not, wit

0

Mmarcuschenvoicelatency.hashnode.devAug 10 · 8 min read

Two weeks before launch, every turn was green and the call still died

The dashboard was a wall of green. Word error rate under 5 percent. Intent classification at 94 percent on our eval set. Response appropriateness, graded by a rubric we trusted, sitting comfortably in

0

Mmarcuschenvoicelatency.hashnode.devAug 5 · 6 min read

The voice A/B test that picked the worse agent, and won by 4 points

We ran a clean A/B test between two versions of a phone agent. Variant B won by 4 points on our success metric. We shipped B. Two weeks later the escalation rate to human agents had gone up, and the "

0

VKVamsi Krishnablog.vamsiannamreddy.comJul 27 · 2 min read

A clean connector does not make a reliable agent

Your integration tests pass. Every tool returns what it should, schemas validate, errors are structured. The agent still spent forty tool calls going in a circle and then confidently reported a result

3

JVC

Mmarcuschenvoicelatency.hashnode.devJul 26 · 5 min read

Our voice agent scored 91 percent. The callers still hung up angry.

Two weeks after we launched the support line, the dashboard was the color you want. Task-success rate: 91 percent. The agent booked the appointment, reset the password, quoted the balance. Green acros

0

Mmayaanderssonllmasajudge.hashnode.devJul 22 · 8 min read

An LLM judge is a biased instrument, not a measurement

Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Variant B won. Same model, same judge, same test set. Th

0

Mmayaanderssonllmasajudge.hashnode.devJul 21 · 8 min read

Your eval dashboard has 30 metrics. When one "moves," that is usually arithmetic, not a regression.

Here is the ritual. You ship a prompt change, rerun the eval suite, and open the dashboard. Thirty numbers sit there: faithfulness, answer relevance, context precision, toxicity, latency-adjusted qual

0

DSDarsh Shahfreecodecamp.orgJul 17 · 12 min read

How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python

In this tutorial, I'll show you how to evaluate a local AI agent with a simple, repeatable evaluation harness. The harness runs the agent against a set of test cases, checks the results with both rule

0

#evaluation

Search Hashnode

#evaluation

Explore Hashnode

Trending tags this week

7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)

What a frontier coding agent actually does when the tests are wrong

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much

Two weeks before launch, every turn was green and the call still died

The voice A/B test that picked the worse agent, and won by 4 points

A clean connector does not make a reliable agent

Our voice agent scored 91 percent. The callers still hung up angry.

An LLM judge is a biased instrument, not a measurement

Your eval dashboard has 30 metrics. When one "moves," that is usually arithmetic, not a regression.

How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python