Per-iteration PyTorch Visibility...
summary
Claims per-iteration visibility into PyTorch step timing with no code change, using a Python site package to emit job/iteration indices and forward/backward/data percentages, plus a NiC plugin, eBPF kernel traces, ETH tool/DCGM device telemetry and a probe mesh for oneway delay. Demo runs across four hosts (8 GPUs each), creates PCIe/NiC congestion and a GPU clock-throttle straggler, attributes each root cause, quantifies impact in GPU‑hours, and shows iteration-rate recovery within minutes — all with low overhead and Prometheus-compatible metrics and a per-iteration breakdown.
AI-written summary · about
chapters (8)
- 0:00ML infra failures intro
- 1:49Engineer visibility challenges
- 3:13Cross-stack observability
- 5:01Demo setup & iteration rate
- 6:26Network congestion detection
- 8:14Hardware issue detection
- 9:38Clockwork workload bridges
- 10:02Q&A
related talks
12mContinuous Profiling for GPUs
21mAuto-Instrumentation for GPU Performance using eBPF
10mLightning Talk: Step-Aligned Telemetry for Distributed PyTorch Training (Time...
10mLightning Talk: Profiling and Memory Debugging Tools for Distributed ML Workloads on GPUs
12mPyTorch Partner Collaborations
25mTorchBench: Guarding the Performance of the PyTorch Ecosystem with Continuous Benchmarking
26mA Year of Observability Across the Toolstack: Breakthroughs, Challenges, Improveme...
01h 00mSystems @Scale Summer Q&A Week 4