The reframing that makes this click is that most distributed bugs are coordination bugs, not numerical ones, and coordination does not care which device the tensor lives on. A rank taking a different branch and hanging a collective is a control-flow bug you can reproduce with two CPU processes on a laptop. Asserting on synchronization rather than on loss is the key discipline here, because loss converging is compatible with gradients not being synced at all - the run just quietly becomes N independent models and nobody notices until the eval numbers are strange. Deliberately failing timeouts is the part most teams skip and later regret, since the default behaviour of a hung collective in CI is a job that burns its full wall clock and reports nothing useful.
The reframing that makes this click is that most distributed bugs are coordination bugs, not numerical ones, and coordination does not care which device the tensor lives on. A rank taking a different branch and hanging a collective is a control-flow bug you can reproduce with two CPU processes on a laptop. Asserting on synchronization rather than on loss is the key discipline here, because loss converging is compatible with gradients not being synced at all - the run just quietly becomes N independent models and nobody notices until the eval numbers are strange. Deliberately failing timeouts is the part most teams skip and later regret, since the default behaviour of a hung collective in CI is a job that burns its full wall clock and reports nothing useful.