Scaling the Batch and the Cluster: Gradient Accumulation and 1D Data Parallelism

Part 3 of 4 — Scaling LLM Training. Part 1 mapped where memory goes; Part 2 freed some of it by trading FLOPs for HBM. But recomputation only helps within a single device. To reach a large effective b