Mmayaanderssoninllmasajudge.hashnode.dev·1d ago · 20 min readYour eval pass rate is 98 percent. Your confidence interval is probably wrong.TL;DR. Almost every eval harness reports a pass rate with an error bar, and almost every one of those error bars comes from the normal approximation: p̂ plus or minus 1.96 times the square root of p̂(00
Mmayaanderssoninllmasajudge.hashnode.dev·3d ago · 18 min readComparing Two Eval Runs by Their Average Pass Rate Is the Wrong TestTL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B passes 74.0 percent, and you conclude B is better. That reasoning throws away the one fact that matt00
Mmayaanderssoninllmasajudge.hashnode.dev·Jul 8 · 7 min readOne average eval score was hiding two different failure modesA mean faithfulness of 0.75 sounds like a model that is usually right and occasionally slips. Mine was near-perfect on half the data and near-zero on the other half, and 0.75 described neither slice. 00
Mmayaanderssoninllmasajudge.hashnode.dev·Jul 7 · 8 min readYour LLM-as-judge has a position bias you are not measuringIf your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the order, every win-rate you report is contaminated by which slot you happened to put each answer in. The00
Mmayaanderssoninllmasajudge.hashnode.dev·Jul 1 · 7 min readI reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.The industry has converged on a definition of "operator-ready" that is measurable, deployable, and wrong. The most cited frameworks, Anthropic's "Building Effective Agents" (December 2024), Hamel Husa00
