Mmayaanderssoninllmasajudge.hashnode.dev·5d ago · 8 min readAn LLM judge is a biased instrument, not a measurementLast month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Variant B won. Same model, same judge, same test set. Th00
Mmayaanderssoninllmasajudge.hashnode.dev·6d ago · 8 min readYour eval dashboard has 30 metrics. When one "moves," that is usually arithmetic, not a regression.Here is the ritual. You ship a prompt change, rerun the eval suite, and open the dashboard. Thirty numbers sit there: faithfulness, answer relevance, context precision, toxicity, latency-adjusted qual00
Mmayaanderssoninllmasajudge.hashnode.dev·Jul 16 · 20 min readYour eval pass rate is 98 percent. Your confidence interval is probably wrong.TL;DR. Almost every eval harness reports a pass rate with an error bar, and almost every one of those error bars comes from the normal approximation: p̂ plus or minus 1.96 times the square root of p̂(00
Mmayaanderssoninllmasajudge.hashnode.dev·Jul 14 · 18 min readComparing Two Eval Runs by Their Average Pass Rate Is the Wrong TestTL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B passes 74.0 percent, and you conclude B is better. That reasoning throws away the one fact that matt00
Mmayaanderssoninllmasajudge.hashnode.dev·Jul 8 · 7 min readOne average eval score was hiding two different failure modesA mean faithfulness of 0.75 sounds like a model that is usually right and occasionally slips. Mine was near-perfect on half the data and near-zero on the other half, and 0.75 described neither slice. 00
