Resilience for AI Workloads at Scale: The Fast and the Finicky!
summary
Claims that NIC and link flaps (optical transceivers overheating/dust) make RDMA fabrics crash large LLM jobs and waste checkpoint time; demos use a software probe mesh to measure per-QP one-way delays and show slow QPs (up to ~150 µs) causing 30%+ of job time and throughput drops from ~92 Gb/s to 44–50 Gb/s under contention. Demo of a link flap kills a job; an auto-recovery preserves job progress case continues with modest throughput loss. link flap causes job kill and probe mesh measures Q-pair one-way delay.
AI-written summary · about
chapters (9)
- 0:00AI workload challenges
- 2:27AI workload layers and network
- 6:21Backend network challenges
- 9:35Field challenges for AI networks
- 10:57Demo: Measuring network delays
- 17:13Demo: Simulating link flaps
- 20:47Solutions for resilient AI workloads
- 21:08Demo: Handling network contention
- 24:04Key takeaways and hiring
related talks
19mAI/ML Networking Challenges: The Fast and the Finicky!
20mTurbocharging AI/ML workloads: Revving Up Speed and Resilience
30mThe Gashlycrumb Tinies of AI Networking You Must Know (or Languish!)
22mLive Q&A Session #1 – Srilakshmi Adusumali
02h 53mPreview Sessions
01mThe Future of AI Networking Starts Here
14mUnderstanding RDMA Microarchitecture Resources for Performance Isolation
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective