Activation Recomputation and the HFU/MFU Trap

Part 2 of 4 — Scaling LLM Training. In Part 1 we saw that activation memory — not model state — is usually what drives you out of HBM, and that the attention-score term scales with the square of seque