The reframing that makes this click is that most distributed bugs are coordination bugs, not numerical ones, and coordination does not care which device the tensor lives on. A rank taking a different branch and hanging a collective is a control-flow bug you can reproduce with two CPU processes on a laptop. Asserting on synchronization rather than on loss is the key discipline here, because loss converging is compatible with gradients not being synced at all - the run just quietly becomes N independent models and nobody notices until the eval numbers are strange. Deliberately failing timeouts is the part most teams skip and later regret, since the default behaviour of a hung collective in CI is a job that burns its full wall clock and reports nothing useful.