TTTerry Taointaot.hashnode.dev·Feb 8 · 3 min readGrokking of Transformer on Modular AdditionBackground Grokking is a phenomenon where a model quickly achieves near-perfect training accuracy (memorization), while validation accuracy remains near chance for a long time, and then later transitions sharply to strong generalization after extende...00
TTTerry Taointaot.hashnode.dev·Jan 24 · 4 min readGrokking of MLP on Modular AdditionBackground Grokking is a phenomenon where a model quickly achieves near-perfect training accuracy (memorization), while validation accuracy remains near chance (or even worse-than-chance) for a long time, and then later transitions sharply to strong ...00
