The reframing that makes this click is that most distributed bugs are coordination bugs, not numerical ones, and coordination does not care which device the tensor lives on. A rank taking a different branch and hanging a collective is a control-flow bug you can reproduce with two CPU processes on a laptop. Asserting on synchronization rather than on loss is the key discipline here, because loss converging is compatible with gradients not being synced at all - the run just quietly becomes N independent models and nobody notices until the eval numbers are strange. Deliberately failing timeouts is the part most teams skip and later regret, since the default behaviour of a hung collective in CI is a job that burns its full wall clock and reports nothing useful.
Thanks, this captures the main idea really well. The biggest takeaway for me was that many distributed failures are coordination problems, not GPU problems. The DDP synchronization check and deliberate timeout failure were especially useful because they test behavior that can otherwise fail silently or waste a lot of CI time.
The timeout test also needs to assert cleanup, not only that the parent test returned an error. A hung rank can outlive the assertion and interfere with the next test through occupied ports or lingering child processes, making a coordination test look intermittently broken for unrelated reasons.
I would end the failure case by checking that all ranks exited and that the next small collective can start cleanly. That makes the deliberate timeout a test of recovery as well as detection, which is the behavior CI needs after the expected failure.
Ahmet Özel
The reframing that makes this click is that most distributed bugs are coordination bugs, not numerical ones, and coordination does not care which device the tensor lives on. A rank taking a different branch and hanging a collective is a control-flow bug you can reproduce with two CPU processes on a laptop. Asserting on synchronization rather than on loss is the key discipline here, because loss converging is compatible with gradients not being synced at all - the run just quietly becomes N independent models and nobody notices until the eval numbers are strange. Deliberately failing timeouts is the part most teams skip and later regret, since the default behaviour of a hung collective in CI is a job that burns its full wall clock and reports nothing useful.