The Gashlycrumb Tinies of AI Networking You Must Know (or Languish!)
summary
Demos show an NCCL all‑reduce hitting 1.47 Tbps on 16×100 Gb NICs until a second cross‑rack job cuts per‑NIC throughput from ~92 Gb to 50–30 Gb, driven by queue‑pair latencies spiking from microseconds into the 50–150 μs range and halving job throughput. A mitigation that aligns one‑way delays to ~16 μs restores throughput near the max. Flapping a NIC kills a job and reviving the port does not recover it; checkpointing (~30 min) or tools like TorchFT are required to limit lost work.
AI-written summary · about
chapters (25)
- 0:00AI networking failures intro
- 1:02Introduction
- 2:27AI training at scale
- 3:09AI workload layers
- 4:34Poll on stack
- 6:44PyTorch training demo
- 7:53Processes and GPU usage
- 9:17Parallelism strategies
- 10:21NCCL’s role
- 11:48All reduce and stragglers
- 13:14Network contention
- 14:17All reduce throughput demo
- 16:46Reducing one-way delays
- 17:06Back-end vs front-end
- 18:28RDMA NIC latencies demo
- 19:51RDMA kernel bypass
- 20:56InfiniBand vs RoCE v2
- 21:59Network link flapping
- 23:24NIC flapping impact demo
- 24:49Recovering from flapping
- 25:50Checkpointing failures
- 26:31GPU failures impact
- 27:15Checkpointing training
- 28:41Fault tolerance strategies
- 29:02Key takeaways
related talks
25mResilience for AI Workloads at Scale: The Fast and the Finicky!
19mAI/ML Networking Challenges: The Fast and the Finicky!
20mTurbocharging AI/ML workloads: Revving Up Speed and Resilience
14mHigh-throughput and Flexible Host Networking for Accelerated Computing
30mNicPic: Scalable and Accurate End-Host Rate Limiting
19mEnabling Programmable Transport Protocols in High Speed NICs
21mReexamining Direct Cache Access to Optimize I/O Intensive Applications for Multi
02h 53mPreview Sessions