PyTorch Symmetric Memory + NCCL Device APIs: A New Path Towards Multi-GP...
summary
A new stack enables direct GPU-to-GPU memory access without CPU involvement using PyTorch Symmetric Memory for symmetric tensors and NCCL device APIs (LSA, multi-mem, GEN/GIN) for in-kernel NVLink and GPU-initiated networking. Users allocate symmetric tensors via torch.distributed.symmetric_memory and write fused kernels; an AllGather-Matmul overlap using CUDA copy-engine shows a measured 14% throughput gain and NICO EP (open source) matches DeepSpeed MoE performance and edges it at larger scale.
AI-written summary · about
chapters (13)
- 0:00PyTorch Symmetric Memory + NCCL intro
- 1:27Distributed workload challenges
- 3:12Symmetric memory explained
- 4:57NCCL Device API overview
- 7:05LSA and multi-mem API
- 8:48GEN network API
- 10:33CUTLASS integration plans
- 11:57NCCL EP for MOE tokens
- 13:42NCCL EP performance status
- 14:27Symmetric memory features
- 16:32Collectives in PyTorch memory
- 17:57Copy engine collective
- 19:01Q&A
related talks
34mSymmetric Memory in PyTorch: 10x Faster GPU Communication for AI
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
19mSymmetric Memory: A New Paradigm for Programming Distributed AI
01h 17m📚 Advanced NCCL Tutorial: Beyond Collectives
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
24mTraining large scale models using PyTorch
20mLegion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale...
15mColocating ML Inference and Training with Fast GPU Memory Handover