Good framing of agent evals as a statistics problem. One suggestion on the latency section: agent latency and token cost are usually heavily right-skewed, so mean and Z-scores can mislead because a few slow runs inflate the standard deviation and hide themselves. Percentiles (p50/p95/p99) or a robust measure like median absolute deviation tend to be safer for flagging outliers. For quality or pass-rate metrics with small sample sizes, a bootstrap or Wilson confidence interval helps avoid reading noise as regression in a CI gate. Since the same prompt can score differently across runs, how would you set the gate threshold: a fixed pass-rate floor, or a comparison against the previous version's distribution?
AI Developer
Good framing of agent evals as a statistics problem. One suggestion on the latency section: agent latency and token cost are usually heavily right-skewed, so mean and Z-scores can mislead because a few slow runs inflate the standard deviation and hide themselves. Percentiles (p50/p95/p99) or a robust measure like median absolute deviation tend to be safer for flagging outliers. For quality or pass-rate metrics with small sample sizes, a bootstrap or Wilson confidence interval helps avoid reading noise as regression in a CI gate. Since the same prompt can score differently across runs, how would you set the gate threshold: a fixed pass-rate floor, or a comparison against the previous version's distribution?