How to Correct an LLM Judge's Pass Rate With 100 Human Labels
TL;DR: The pass rate a grading model reports is part your product and part the grader's own mistakes. Measure those mistakes on a small set a person has checked, and you can subtract them back out. En
Responses
Markdown supported · ⌘↵ to post
AI Developer
build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results.
The Rogan-Gladen step divides by TPR+TNR-1, so the uncertainty in TNR matters most, and a random 100-label sample at a ~90% pass rate leaves only about 10 truly bad outputs to estimate it from. It may be worth stratifying the labeling (oversample judge-Pass items, or known-bad ones) and bootstrapping a confidence interval on the corrected rate. Also, TPR/TNR are only valid for that judge model and prompt version, so any judge change would seem to need a fresh label check. Have you looked at how stable they are across prompt revisions?