Zero Bubble Pipeline Parallelism. (arXiv:2401.10241v1 [cs.DC])

Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work, we introduce a scheduling strategy that, to our knowledge, is the fir...