A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound