Discussion on "A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides"

mayaandersson · 2026-08-17T17:40:10.631Z

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound

Discussion on "A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides" | Hashnode

Search Hashnode

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

Responses