Efficient Direct-Connect Topologies for Collective Communications
summary
Presents a concrete pipeline that synthesizes direct‑connect topologies and schedules, starting from small optimal bases and using line graph expansion and other graph transforms to grow to 1,024+ nodes while keeping degree bounded and diameter growing ~log n. Introduces BFB scheduling (breadth‑first broadcast) that reduces schedule synthesis to a polynomial‑time LP, generates schedules for thousands of nodes in minutes, runs on a 12× A100 optical testbed, yields >40% speedups in mixed training vs shifted ring and stays within 5% of the theoretical lower bound.
AI-written summary · about
chapters (10)
- 0:00Collective communication in ML
- 1:20Optical circuit networks challenges
- 2:44Traditional topologies limitations
- 3:28Low-diameter topologies with schedules
- 4:11Topology scaling expansion
- 6:18BFB scheduling for reduce
- 9:07BFB performance guarantees
- 10:10Topology finder for matching
- 11:34Experimental validation on testbeds
- 13:41Conclusion and future
related talks
15mForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics
17mTopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed...
14mHeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous
21mThe Evolution of Traffic Routing in a Streaming World
13mContracting Wide-area Network Topologies to Solve Flow Problems Quickly
44mTutorial on List Decodable Coding Schemes and Interactive Communication in Networks (Part 2 of 2)
22mUnderstanding Lifecycle Management Complexity of Datacenter Topologies
16mAccelerating Distributed Graph Learning by Using Collaborative In-Network...