1d ago · 4 min read · On a shared GPU host, the kernel knows which container generated which event. The trace stays separable as long as cgroup_id rides along. TL;DR On a shared GPU host (think K8s with three GPU pods, or
Join discussion
3d ago · 7 min read · TL;DR In a distributed training job, every node can look healthy on its own dashboard while throughput across the job quietly drops. The cause is almost never visible per host, because the signal is r
Join discussionMay 29 · 7 min read · TL;DR A GPU that reports 97% utilization can still be the slowest part of a training step, and the reason usually lives outside the GPU: a CPU scheduler preemption, a driver-level allocation, a collec
Join discussionMay 28 · 7 min read · TL;DR After del tensor; torch.cuda.empty_cache(), PyTorch’s caching allocator still holds 53.7 MB that it won’t release. We traced the CUDA Runtime and Driver APIs with eBPF uprobes to see exactly wh
Join discussion
May 26 · 6 min read · Three eBPF patterns hyperscalers run in production today, mapped to the equivalent patterns on the GPU plane that nobody runs in production yet. TL;DR GitHub recently disclosed using eBPF in productio
Join discussion
May 21 · 6 min read · TL;DR PyTorch’s DataLoader can be 50-124x slower than direct tensor indexing for in-memory GPU workloads. We reproduced a real PyTorch issue on an RTX 4090 and traced every CUDA API call and Linux ke
Join discussion
May 19 · 8 min read · Per-host overhead multiplied across N hosts, vs. one kernel-level instrumentation per host. The math at fleet scale is harder to argue with than the marketing one. TL;DR Wolfe Research disclosed this
Join discussion
