Debugging Silent Model Divergence with Gradient Monitoring

Thanks, and this is the question the post deliberately left open, so let me answer it properly. Answer: Neither, at least not on raw norms. Absolute thresholds are a tripwire, not a monitor: I keep NaN/inf and an insane ceiling, nothing else.

The scale-free version is what survives the drift you're describing. Per layer, ||grad|| / ||weight||, or better on Adam, the post-optimizer ||lr * update|| / ||weight||, since Adam rescales gradients and the raw norm stops telling you how far the weights actually moved. Then a rolling median plus MAD over a trailing window, and alert at k MADs out, so the baseline drifts with training instead of needing a retune every time you change depth or schedule.

For your "a few layers blow up or die" specifically: max(layer ratio) / median(layer ratio). A global norm averages precisely that away. And if you already clip, the fraction of steps hitting the ceiling is a free early warning.