Tag feed

#evaluation

73 posts1 followers

Explore Hashnode

Alternatives

Trending tags this week

Mmayaanderssonllmasajudge.hashnode.devAug 25 · 12 min read

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much

Almost every RAG eval writeup I read, including several I have recommended, uses the same mental model. The system is two stages in series. Retrieval either finds the right context or it does not, wit

0

Mmarcuschenvoicelatency.hashnode.devAug 10 · 8 min read

Two weeks before launch, every turn was green and the call still died

The dashboard was a wall of green. Word error rate under 5 percent. Intent classification at 94 percent on our eval set. Response appropriateness, graded by a rubric we trusted, sitting comfortably in

0

Mmarcuschenvoicelatency.hashnode.devAug 5 · 6 min read

The voice A/B test that picked the worse agent, and won by 4 points

We ran a clean A/B test between two versions of a phone agent. Variant B won by 4 points on our success metric. We shipped B. Two weeks later the escalation rate to human agents had gone up, and the "

0

VKVamsi Krishnablog.vamsiannamreddy.comJul 27 · 2 min read

A clean connector does not make a reliable agent

Your integration tests pass. Every tool returns what it should, schemas validate, errors are structured. The agent still spent forty tool calls going in a circle and then confidently reported a result

3

JVC

Mmarcuschenvoicelatency.hashnode.devJul 26 · 5 min read

Our voice agent scored 91 percent. The callers still hung up angry.

Two weeks after we launched the support line, the dashboard was the color you want. Task-success rate: 91 percent. The agent booked the appointment, reset the password, quoted the balance. Green acros

0

Mmayaanderssonllmasajudge.hashnode.devJul 22 · 8 min read

An LLM judge is a biased instrument, not a measurement

Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Variant B won. Same model, same judge, same test set. Th

0

Mmayaanderssonllmasajudge.hashnode.devJul 21 · 8 min read

Your eval dashboard has 30 metrics. When one "moves," that is usually arithmetic, not a regression.

Here is the ritual. You ship a prompt change, rerun the eval suite, and open the dashboard. Thirty numbers sit there: faithfulness, answer relevance, context precision, toxicity, latency-adjusted qual

0

DSDarsh Shahfreecodecamp.orgJul 17 · 12 min read

How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python

In this tutorial, I'll show you how to evaluate a local AI agent with a simple, repeatable evaluation harness. The harness runs the agent against a set of test cases, checks the results with both rule

0

TMThamu Mnyulwathamu.hashnode.devJul 14 · 11 min read

Evaluer la GenAI avec mlflow.genai.evaluate() : au-dela de l'accuracy

La premiere erreur dans l'evaluation des systemes GenAI consiste a traiter la sortie comme une etiquette de classification. Cela fonctionne pour des taches de machine learning deterministes. Cela ne f

0

TMThamu Mnyulwathamu.hashnode.devJul 14 · 10 min read

GenAI Evaluation with mlflow.genai.evaluate(): Beyond Accuracy

The first mistake teams make when they evaluate GenAI systems is treating the output like a classification label. That works for deterministic machine learning tasks. It does not work for a retrieval-

0

#evaluation

Search Hashnode

#evaluation

Explore Hashnode

Trending tags this week

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much

Two weeks before launch, every turn was green and the call still died

The voice A/B test that picked the worse agent, and won by 4 points

A clean connector does not make a reliable agent

Our voice agent scored 91 percent. The callers still hung up angry.

An LLM judge is a biased instrument, not a measurement

Your eval dashboard has 30 metrics. When one "moves," that is usually arithmetic, not a regression.

How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python

Evaluer la GenAI avec mlflow.genai.evaluate() : au-dela de l'accuracy

GenAI Evaluation with mlflow.genai.evaluate(): Beyond Accuracy