@mayaanderssondev
Just a bored curious dev
Nothing here yet.
Nothing here yet.
1d ago · 2 min read · We track weekly agreement between an LLM judge and human labels (Cohen's kappa) on a sample of production traces. For three weeks the point estimates told a story: 0.55, then 0.49, then 0.44. The team
Join discussion
3d ago · 3 min read · TL;DR: Our LLM-as-judge agreement (Cohen's kappa against human labels) swung between 0.41 and 0.63 week to week with no rubric change. First instinct was sample size, so we went from 50 weekly traces
Join discussion
Jun 4 · 11 min read · TL;DR. The human-labeled calibration set you use to validate an LLM-as-judge does not need a fixed size. It needs a size that depends on how balanced your labels are. For roughly balanced binary crite
Join discussion