Live Q&A Session #2 – Adi Gangadi
summary
Panelists describe concrete fixes: network metrics (PFC, ECN, buffer and port utilization) reveal RX pauses that hampered zero-copy alltoall; switching from explicit to transparent buffer registration via a PyTorch cache cuts registration latency from milliseconds per buffer; a custom middle‑ground transport is built alongside vendor libraries and will be open sourced; fault tolerant training shrinks effective failure domains from ~100k to ~4k GPUs but trades reproducibility and requires offload, selective retransmit logic, and careful replica sizing.
AI-written summary · about
chapters (11)
- 0:00Performance bottleneck identification
- 0:48Observability approaches
- 2:54Initialization scaling vs fault-tolerant training
- 3:56Software vs hardware load balancing
- 5:20Custom vs open-source communication libraries
- 7:30Reducing interruptions in fault-tolerant training
- 9:39Fault-tolerant training challenges
- 11:27Tracking parallelism configurations
- 12:53NCCL zero-copy limitations
- 15:42Scaling challenges
- 20:37Closing remarks
related talks
33mLive Q&A Session 2
27mLive Q&A Session 1
19mAI/ML Networking Challenges: The Fast and the Finicky!
24mSponsored Session: PyTorch Distributed and Fault Tolerance
20mLive Q&A 3 with Speakers
26mWeek 1 Live Q&A
34mPanel Discussion: Training and Inference at Planet Scale
38mBeyond Kubernetes: Adapting To Spe...