Tag feed

#evaluation

81 posts1 followers

Explore Hashnode

Alternatives

Trending tags this week

YLYaoshen Luoluoyaoshen.hashnode.devSep 29 · 7 min read

Benchmarking Real Work - Case 01: How I Built a Voice Agent Benchmark from Real Customer Failures

Customer complaints are not the problem definition; they are the signal. This post captures a real-world case of agent benchmarking: how I built v1 of our benchmark from real customer failure logs—wi

1

L

Mmarketeworldlive.hashnode.devSep 28 · 1 min read

On our frozen InvoiceHub runs, R@5 picks V_push_4_5 (1.00) — nDCG@5 picks V_vector (0.44)

Author 116 InvoiceHub FAQ chunks × 15 queries, three frozen MiniLM ranking variants on the same graded gold (seed 20260928, API $0). Score Recall@5/10 (binary = grade ≥ 1) beside nDCG@5/10 (graded 0/1

0

Mmarketeworldlive.hashnode.devSep 28 · 1 min read

On our frozen MiniLM InvoiceHub run, κ(A,B) 0.82 but 30% label noise lifted apparent P@5 from 0.23 to 0.39

Author 116 InvoiceHub FAQ chunks × 15 queries, one frozen MiniLM top-5 / pool-20 run (seed 20260928, API $0). Dual-labeled the pool (Annotator-A vs B), then flip-noise 0/10/20/30% vs A. Retrieval list

0

Mmarketeworldlive.hashnode.devSep 28 · 1 min read

On our PulseBoard FAQ, template synth hit R@5 0.87 vs handwritten 0.47 — lift +0.40 at 43% answer-leakage

Author 120 PulseBoard chunks (80 FAQ + 40 near-dupe distractors) × fixed MiniLM index (seed 20260928, API $0). Same index, two query sets: 15 handwritten hard paraphrases vs 30 template-synth from chu

0

Mmarketeworldlive.hashnode.devSep 28 · 1 min read

Lost-in-middle on our 12 needles: end 12/12, begin/mid 0/12 (not a U-curve copy)

Canonical: Lost-in-middle context position self-test on eWorld.liveCheck date: 2026-09-28 PT · API spend $0 Author HarborMesh fiction: 12 gold needles, K=10 (1 relevant + 9 distractors), same needle a

0

Mmarketeworldlive.hashnode.devSep 27 · 1 min read

RAG citation checklist: hand Faithfulness 0.8 on a mini corpus

RAG citation checklist: hand Faithfulness 0.8 on a mini corpus Canonical: RAG citation groundedness checklist on eWorld.liveCheck date: 2026-09-26 PT · Human desk labels · no LLM judge API Built a 5-c

0

Mmarketeworldlive.hashnode.devSep 27 · 1 min read

Same FAQ set on Promptfoo vs Inspect (Braintrust hosted UNVERIFIED)

Same FAQ set on Promptfoo vs Inspect (Braintrust hosted UNVERIFIED) Canonical: Agent eval harness compare on eWorld.liveCheck date: 2026-09-26 PT · Stub path · $0 API spend We ran one author 12-item F

0

DGDebashish Ghosalpragmatic-engineer.hashnode.devSep 16 · 7 min read

7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)

You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on. Three weeks later you find out the number was never

0

SDSagnik Deystop-gate.hashnode.devSep 10 · 12 min read

What a frontier coding agent actually does when the tests are wrong

Everyone building in this space cites ImpossibleBench: frontier models exploit test cases in a large fraction of runs where the specification and the tests conflict. Far fewer people run a detector ag

3

A

Mmayaanderssonllmasajudge.hashnode.devAug 25 · 12 min read

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much

Almost every RAG eval writeup I read, including several I have recommended, uses the same mental model. The system is two stages in series. Retrieval either finds the right context or it does not, wit

0

#evaluation

Search Hashnode

#evaluation

Explore Hashnode

Trending tags this week

Benchmarking Real Work - Case 01: How I Built a Voice Agent Benchmark from Real Customer Failures

On our frozen InvoiceHub runs, R@5 picks V_push_4_5 (1.00) — nDCG@5 picks V_vector (0.44)

On our frozen MiniLM InvoiceHub run, κ(A,B) 0.82 but 30% label noise lifted apparent P@5 from 0.23 to 0.39

On our PulseBoard FAQ, template synth hit R@5 0.87 vs handwritten 0.47 — lift +0.40 at 43% answer-leakage

Lost-in-middle on our 12 needles: end 12/12, begin/mid 0/12 (not a U-curve copy)

RAG citation checklist: hand Faithfulness 0.8 on a mini corpus

Same FAQ set on Promptfoo vs Inspect (Braintrust hosted UNVERIFIED)

7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)

What a frontier coding agent actually does when the tests are wrong

Dividing your RAG score by retrieval recall overstates your generation quality, and here is by how much