Tag feed

#llm-evaluation

23 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

MSManu Shuklaecorpit.hashnode.dev3d ago · 14 min read

LLM tool-use reliability in 2026: how to evaluate models for AI agents

LLM tool-use reliability in 2026: how to evaluate models for AI agents Summary. The benchmark that predicts whether an AI agent survives production is not the one most teams check. The Berkeley τ-benc

0

KNKartik N V J Kkartiknvjk.hashnode.devJul 28 · 7 min read

How I generate LLM test cases that actually catch bugs

I got tired of hand-writing test cases for our agent, so I did the obvious 2026 thing: I pointed an LLM at our docs and asked it to generate them. In an afternoon I had a few thousand. I felt incredib

0

AAAwais Ahmadblog.itsahmadawais.comJul 28 · 8 min read

LLM Evaluation for Engineers: Building Reliable AI Applications

When building traditional software, engineers write tests to make sure code works correctly. Every time you change a feature or update a package, automated tests run to catch bugs before they reach re

0

KNKartik N V J Kkartiknvjk.hashnode.devJul 23 · 7 min read

Evaluating an intent classifier: what I check beyond accuracy

I built the intent router that sits at the front of our support agent. It reads what a user typed and decides which pipeline handles it. I tested it on a nice balanced set, got 92 percent accuracy, an

0

AHAkash Hadagali Persettiakashpersetti.hashnode.devJul 20 · 9 min read

My eval leaderboard published a negative retry count

I was looking at the EvalBench dashboard a few weeks ago and one cell stopped me: Retries to valid 0.125 95% CI -0.035 – 0.285 n=40 A negative number of retries. You cannot retry a request negative-p

0

ACAlexander Codesalexandercodes.hashnode.devJul 14 · 9 min read

How to Evaluate LLM Answers with the dbt Semantic Layer

An AI assistant can produce a perfectly valid MetricFlow query, and still answer the wrong question. It might choose the wrong metric, dimension, filter, or time grain. The query runs, the result look

0

Mmayaanderssonllmasajudge.hashnode.devJul 1 · 7 min read

I reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.

The industry has converged on a definition of "operator-ready" that is measurable, deployable, and wrong. The most cited frameworks, Anthropic's "Building Effective Agents" (December 2024), Hamel Husa

0

Mmayaanderssonllmasajudge.hashnode.devJun 29 · 7 min read

We added synthetic data to our eval set. The pass rate rose, and so did our production incidents

We needed a bigger eval set, so we generated one. A model wrote a few thousand test cases that looked like our traffic, we scored against them, the pass rate went up, and we felt good. Then production

0

VIVictor Isuovictorisuo.hashnode.devJun 27 · 11 min read

Why I Built a Separate Evaluation System Before Trusting My Own AI

The Industrial AI Copilot had been running for three weeks when I realised I had no idea whether it was actually working. Not in the catastrophic sense — no exceptions, no crashes, no hallucinations I

0

Mmayaanderssonllmasajudge.hashnode.devJun 25 · 5 min read

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Most LLM-as-judge comparisons rank tools by which one gives you a number fastest. That is the wrong axis. A judge you have not validated against human labels is not a measurement, it is a vibe with a

0

#llm-evaluation

Search Hashnode

#llm-evaluation

Explore Hashnode

Trending tags this week

LLM tool-use reliability in 2026: how to evaluate models for AI agents

How I generate LLM test cases that actually catch bugs

LLM Evaluation for Engineers: Building Reliable AI Applications

Evaluating an intent classifier: what I check beyond accuracy

My eval leaderboard published a negative retry count

How to Evaluate LLM Answers with the dbt Semantic Layer

I reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.

We added synthetic data to our eval set. The pass rate rose, and so did our production incidents

Why I Built a Separate Evaluation System Before Trusting My Own AI

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.