Mmayaanderssoninllmasajudge.hashnode.dev·9h ago · 4 min readSix prompt-optimization frameworks: what matters when you run them on the same taskTL;DR: I ran six prompt-optimization frameworks against the same task and the same eval metric over a few weeks (DSPy, GEPA, TextGrad, agent-opt, Arize Prompt Learning, and MLflow's optimizer). They a00
Mmayaanderssoninllmasajudge.hashnode.dev·1d ago · 9 min readA judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decidesTL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound00
Mmayaanderssoninllmasajudge.hashnode.dev·6d ago · 5 min readRun forty experiments against one eval set and you will find an improvement that is not thereYou iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an00
Mmayaanderssoninllmasajudge.hashnode.dev·Aug 11 · 9 min readI read the metric libraries of five widely-used eval tools. The metric was never the hard part.Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it qu00
Mmayaanderssoninllmasajudge.hashnode.dev·Aug 7 · 10 min readYour eval monitor fired on four days this week. At your sample size, that was the most likely countA team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate00
