Lightning Talk: Profiling and Memory Debugging Tools for Distributed ML Workloads on GPUs
summary
Overview of concrete debugging steps and outcomes for PyTorch GPU memory: introduces Memory Snapshot that records allocation stacks and produces time-series plots (drag a file into pyto.org/memory viz) and the Reference Cycle Detector that emits object-graph warnings when tensors survive GC. Shows a reproducible bug where GC must run every 100 iterations to reclaim cycle-held tensors, and fixing reference cycles cuts peak GPU usage and reduces fragmentation. Also mentions Profiler/Kineto, Holistic Trace Analysis and Dynolog as tracing and on-demand tools.
AI-written summary · about
chapters (9)
- 0:00Introduction to GPU issues
- 0:39Memory snapshot tool
- 1:47Training example analysis
- 2:34Debugging a memory bug
- 3:42Memory profiler integration
- 4:24Understanding reference cycles
- 6:35Complex reference cycles
- 7:37Reference cycle detector
- 8:38Conclusion and resources
related talks
08mLightning Talk: GPU.x - Dynamic GPU Fraction and Sharing on ML Mainstream Framework
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
45mLIGHTNING TALKS
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
17mLightning Thunder: Supercharged PyTorch for Modern Hardware
07mLightning Talk: Slash LLM Cold-Start Times by Pre-distributing GPU...
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures