Agreed on verbosity bias. One cheap extra check on your human-graded set: correlate the judge's scores with answer length. If that correlation is stronger than the one in the human scores, the bias is showing. The post suggests aiming for roughly 0.85 correlation with human raters and re-checking regularly.
