From Kernel Scheduler to Python Source Line: Tracing a GPU Stall End to End

TL;DR A GPU that reports 97% utilization can still be the slowest part of a training step, and the reason usually lives outside the GPU: a CPU scheduler preemption, a driver-level allocation, a collec