Symmetric Memory: A New Paradigm for Programming Distributed AI
summary
PyTorch symmetric memory implements a one-sided shared-memory API (prototype landed in PyTorch) that lets GPUs read/write peer memory to run low-latency one-shot all-reduce kernels and a previously impossible device-side token-shuffle without CPU sync. It exposes Triton kernels that call symmetric-memory pointers, uses NVISHM for multi-node RDMA so GPUs can originate messages at hundreds of millions of requests per second, and shows these approaches reduce communication latency and enable new overlap of compute and comms.
AI-written summary · about
chapters (7)
- 0:00Introduction to symmetric memory
- 1:51Concept and benefits
- 4:41PyTorch implementation examples
- 5:56Advanced parallelization
- 10:07Writing custom Triton kernels
- 12:15Scaling to multi-node systems
- 16:02Future work and conclusion
related talks
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
34mSymmetric Memory in PyTorch: 10x Faster GPU Communication for AI
25mPyTorch Symmetric Memory + NCCL Device APIs: A New Path Towards Multi-GP...
24mSponsored Session: PyTorch Distributed and Fault Tolerance
14mSYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems
15mColocating ML Inference and Training with Fast GPU Memory Handover
58mKeynote: Enabling the AI Revolution: Next-Generation Memory Solutions for AI Computing(…)
45mBeyond Machine Boundaries: Experience with Memory Disaggregation