Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

Gradient compression has surfaced as a key technique to address the challenge of communication efficiency in distributed learning. In distributed deep learning, however, it is observed that gradient distributions are heavy-tailed, with outliers signi...