The batch-versus-serving contention row is the one I keep running into: a training job's checkpoint burst landing on the same path as a latency-sensitive inference endpoint. Separate node pools help, but the same contention shows up at the network level too, so the checkpoint traffic needs its own lane, not just its own nodes. The gang-scheduling point is a good one as well — partial placement holding expensive GPUs idle is easy to miss until the bill arrives. Do teams you work with actually checkpoint to object storage on a fixed interval, or do most still rely on local disk and hope the job survives?
OneLifeVibe
The batch-versus-serving contention row is the one I keep running into: a training job's checkpoint burst landing on the same path as a latency-sensitive inference endpoint. Separate node pools help, but the same contention shows up at the network level too, so the checkpoint traffic needs its own lane, not just its own nodes. The gang-scheduling point is a good one as well — partial placement holding expensive GPUs idle is easy to miss until the bill arrives. Do teams you work with actually checkpoint to object storage on a fixed interval, or do most still rely on local disk and hope the job survives?