I had a nightly gate go red because the judge scored the same reply 0.7 one night and 0.6 the next, with the pass mark at 0.7. Rerunning flaky cases and gating on a pass rate is close to where I ended up. When you build the forty or fifty hand labels, I'd pull a good share from replies near the pass mark, since that's where my verdicts flipped.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
I had a nightly gate go red because the judge scored the same reply 0.7 one night and 0.6 the next, with the pass mark at 0.7. Rerunning flaky cases and gating on a pass rate is close to where I ended up. When you build the forty or fifty hand labels, I'd pull a good share from replies near the pass mark, since that's where my verdicts flipped.