Walking the GPT-1 forward pass in full to pinpoint which pieces got re-engineered later is a great way to teach this, and the pre-norm over post-norm convergence is the cleanest example. Pre-norm won mostly because it keeps gradients stable enough to train deep stacks without the warmup gymnastics post-norm needed. Are you planning to cover why RMSNorm then displaced LayerNorm in most recent models, since that's the next step in the same story?
Walking the GPT-1 forward pass in full to pinpoint which pieces got re-engineered later is a great way to teach this, and the pre-norm over post-norm convergence is the cleanest example. Pre-norm won mostly because it keeps gradients stable enough to train deep stacks without the warmup gymnastics post-norm needed. Are you planning to cover why RMSNorm then displaced LayerNorm in most recent models, since that's the next step in the same story?