The 9,800 versus 200 example makes the statistical point well, and the operational half is worth adding: precision and recall are not just better metrics, they are a business decision in disguise. A missed failure and a false alarm almost never cost the same amount, and until someone writes down the ratio the team is arguing about thresholds with no way to settle it. Once you have that ratio the threshold falls out of it rather than out of taste. The other trap with rare classes is that a single evaluation number is unstable - with 200 positives, recall moves several points because three examples landed differently in the split, so it is worth reporting a confidence interval before anyone celebrates an improvement.
The 9,800 versus 200 example makes the statistical point well, and the operational half is worth adding: precision and recall are not just better metrics, they are a business decision in disguise. A missed failure and a false alarm almost never cost the same amount, and until someone writes down the ratio the team is arguing about thresholds with no way to settle it. Once you have that ratio the threshold falls out of it rather than out of taste. The other trap with rare classes is that a single evaluation number is unstable - with 200 positives, recall moves several points because three examples landed differently in the split, so it is worth reporting a confidence interval before anyone celebrates an improvement.