Grokking of Transformer on Modular Addition

Background Grokking is a phenomenon where a model quickly achieves near-perfect training accuracy (memorization), while validation accuracy remains near chance for a long time, and then later transitions sharply to strong generalization after extende...