Your LLM Judge Agrees With Itself. That Is Not an Eval.

A judge model can give the same verdicts run after run and still be wrong. In a June 2026 preprint on judge reliability [1], Qwen 3 8B reproduced its own scores almost perfectly on MT-Bench, with a te