Your LLM-as-judge has a position bias you are not measuring

If your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the order, every win-rate you report is contaminated by which slot you happened to put each answer in. The