Mmayaanderssoninllmasajudge.hashnode.dev·Aug 25 · 12 min readDividing your RAG score by retrieval recall overstates your generation quality, and here is by how muchAlmost every RAG eval writeup I read, including several I have recommended, uses the same mental model. The system is two stages in series. Retrieval either finds the right context or it does not, wit00
Mmayaanderssoninllmasajudge.hashnode.dev·Aug 18 · 4 min readSix prompt-optimization frameworks: what matters when you run them on the same taskTL;DR: I ran six prompt-optimization frameworks against the same task and the same eval metric over a few weeks (DSPy, GEPA, TextGrad, agent-opt, Arize Prompt Learning, and MLflow's optimizer). They a00
Mmayaanderssoninllmasajudge.hashnode.dev·Aug 17 · 9 min readA judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decidesTL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound00
Mmayaanderssoninllmasajudge.hashnode.dev·Aug 12 · 5 min readRun forty experiments against one eval set and you will find an improvement that is not thereYou iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an00
Mmayaanderssoninllmasajudge.hashnode.dev·Aug 11 · 9 min readI read the metric libraries of five widely-used eval tools. The metric was never the hard part.Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate(), get a number. The pitch works because it is true, and because it qu00
