A 91% eval pass rate shipped our worst regression. We gate on the delta now.
Our CI eval gate failed anything under 90%. A change landed at 91%, passed, and shipped a regression that cost us two days. The absolute threshold was the problem. it cannot tell a stable 91% from a 9
ethanwalkerwrites.hashnode.dev3 min read