Tag feed

#llm-evaluation

19 posts0 followers

Explore Hashnode

Alternatives

ACAlexander Codesalexandercodes.hashnode.dev11h ago · 9 min read

How to Evaluate LLM Answers with the dbt Semantic Layer

An AI assistant can produce a perfectly valid MetricFlow query, and still answer the wrong question. It might choose the wrong metric, dimension, filter, or time grain. The query runs, the result look

0

Mmayaanderssonllmasajudge.hashnode.devJul 1 · 7 min read

I reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.

The industry has converged on a definition of "operator-ready" that is measurable, deployable, and wrong. The most cited frameworks, Anthropic's "Building Effective Agents" (December 2024), Hamel Husa

0

Mmayaanderssonllmasajudge.hashnode.devJun 29 · 7 min read

We added synthetic data to our eval set. The pass rate rose, and so did our production incidents

We needed a bigger eval set, so we generated one. A model wrote a few thousand test cases that looked like our traffic, we scored against them, the pass rate went up, and we felt good. Then production

0

VIVictor Isuovictorisuo.hashnode.devJun 27 · 11 min read

Why I Built a Separate Evaluation System Before Trusting My Own AI

The Industrial AI Copilot had been running for three weeks when I realised I had no idea whether it was actually working. Not in the catastrophic sense — no exceptions, no crashes, no hallucinations I

0

Mmayaanderssonllmasajudge.hashnode.devJun 25 · 5 min read

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Most LLM-as-judge comparisons rank tools by which one gives you a number fastest. That is the wrong axis. A judge you have not validated against human labels is not a measurement, it is a vibe with a

0

KNKartik N V J Kkartiknvjk.hashnode.devJun 16 · 9 min read

How I stopped my RAG CI gate from lying to me

I have learned to distrust a fast green check on a RAG eval. The failure has a shape I now recognize. A PR lands late in the day. The RAG eval runs in 38 seconds and goes green. Twelve hours later the

0

SJShola Jegedefreecodecamp.orgJun 13 · 30 min read

Mastra vs LangChain: Building an AI Agent Pipeline and Analyzing the Data

A week ago, I saw this tweet: I had just shipped SupportMesh, a multi-tenant AI support platform built on Mastra, so I had opinions from production. I liked the .dowhile() loop, the typed step schem

0

AHAdam Hillhwaph.hashnode.devJun 12 · 16 min read

It's LLMs All the Way Down

Generative AI (GenAI) systems and Large Language Models (LLMs) are empowering us to tackle new types of problems and enabling the implementation of smart, autonomous (or semi-autonomous) systems. Howe

0

RYRamhee Yeonramieeee.meJun 11 · 5 min read

[Paper Review] Operationalizing Large Language Models for Clinical Research Data Extraction: Methods, Quality Control, and Governance

The paper elaborates methodologies for LLM text extraction from research papers. It is not only giving methodologies for extraction of meaningful data, but also assessment and choosing right models. 1

0

NPNikhil Pareeknikhil-p-blogs.hashnode.devJun 3 · 6 min read

Tool-calling eval is four problems, not one

I want to start with a trace that still bothers me. An agent fails to book a flight. The model called search_flights with departure_date="next Friday". The endpoint expected an ISO date, returned a 40

0

#llm-evaluation

Search Hashnode

#llm-evaluation

Explore Hashnode

How to Evaluate LLM Answers with the dbt Semantic Layer

I reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.

We added synthetic data to our eval set. The pass rate rose, and so did our production incidents

Why I Built a Separate Evaluation System Before Trusting My Own AI

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

How I stopped my RAG CI gate from lying to me

Mastra vs LangChain: Building an AI Agent Pipeline and Analyzing the Data

It's LLMs All the Way Down

[Paper Review] Operationalizing Large Language Models for Clinical Research Data Extraction: Methods, Quality Control, and Governance

Tool-calling eval is four problems, not one

Trending tags this week