Accelerating PyTorch FSDP Via Overlapping Collectives With In...
summary
Extension of the UCC library and a prototype chain of Nvidia SHArP plus a zerocopy SPIN backend enables overlapping ReduceScatter and Allgather for PyTorch FSDP using in‑network computing and InfiniBand multicast. Implementation runs on 10‑GPU setups (ConnectX NICs, IB quantum switch) and shows microbenchmarked collective latency down ~22.5% and real LLM training throughput up ~10.7% while claiming up to 2x communication reduction in ideal large‑GPU limits. Overlap requires a custom NCCL/UCC/NPI integration and works without user code changes.
AI-written summary · about
chapters (8)
- 0:00PyTorch FSDP intro
- 1:26FSDP backward bottlenecks
- 2:56Reducing backward time
- 4:43In-network computing role
- 6:31PyTorch prototype
- 8:43Performance results
- 10:09Takeaways and future
- 10:56Q&A
related talks
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
09mHyperParallel: A SuperPoD-Aware Distributed Acceleration Library
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
08mPyTorch Distributed: Towards Large Scale Training
15mACCL+: an FPGA-Based Collective Engine for Distributed Applications
16mAccelerating Distributed Graph Learning by Using Collaborative In-Network...
06mMaximizing Programmable DPUs in the Network
10mPyTorch Distributed Data Parallel (DDP)