Mmayaanderssoninllmasajudge.hashnode.dev·18h ago · 8 min readYour LLM-as-judge has a position bias you are not measuringIf your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the order, every win-rate you report is contaminated by which slot you happened to put each answer in. The00
Mmayaanderssoninllmasajudge.hashnode.dev·6d ago · 7 min readI reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.The industry has converged on a definition of "operator-ready" that is measurable, deployable, and wrong. The most cited frameworks, Anthropic's "Building Effective Agents" (December 2024), Hamel Husa00
Mmayaanderssoninllmasajudge.hashnode.dev·Jun 29 · 7 min readWe added synthetic data to our eval set. The pass rate rose, and so did our production incidentsWe needed a bigger eval set, so we generated one. A model wrote a few thousand test cases that looked like our traffic, we scored against them, the pass rate went up, and we felt good. Then production00
Mmayaanderssoninllmasajudge.hashnode.dev·Jun 25 · 5 min readI checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.Most LLM-as-judge comparisons rank tools by which one gives you a number fastest. That is the wrong axis. A judge you have not validated against human labels is not a measurement, it is a vibe with a 00
Mmayaanderssoninllmasajudge.hashnode.dev·Jun 19 · 4 min readI benchmarked 6 prompt-optimization frameworks on the same task. Here is what each one actually optimizes. TL;DR: I ran six prompt-optimization frameworks against the same task and the same eval metric over a few weeks. They are not interchangeable: some are full programming models, some are single search 00
