Constitutional AI in Python: Keep the Soft Labels, Not Just the Winner

An AI judge prefers answer A with probability 0.6. Should the preference model train on 0.6, or should we round it to a hard label of 1? Those choices optimize different objectives. The difference bec