One bad payload took down our kafka consumer group for 45 Minutes

The alert fired in the afternoon: consumer lag climbing on a topic that normally sits close to zero, not a spike - a wall. Messages were arriving, nothing was being processed, and the lag graph looked