One average eval score was hiding two different failure modes

A mean faithfulness of 0.75 sounds like a model that is usually right and occasionally slips. Mine was near-perfect on half the data and near-zero on the other half, and 0.75 described neither slice.