TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
summary
TACCL is a synthesizer that uses communication sketches to prune an MILP search and generate concrete ALLGATHER/ALLREDUCE/ALLTOALL algorithms tuned to DGX‑2 and NDv2 hardware. The pipeline profiles topology, accepts a small sketch (logical links, NVSwitch choices, symmetry), then routes, orders, and batches chunks into an executable. Generated algorithms beat NCCL by up to 6.7x on microbenchmarks and speed end‑to‑end Transformer‑XL/BERT training by 11%–2.3x, demonstrating practical gains.
AI-written summary · about
chapters (9)
- 0:00Introduction to collectives
- 1:32Challenges in communication
- 3:07Current solutions and tackle
- 4:59Tackle workflow and sketches
- 6:24Types of communication sketches
- 8:39Synthesis algorithm stages
- 10:20Algorithm optimization steps
- 11:49Evaluation results
- 13:59Conclusion and Q&A
related talks
14mHeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous
15mAutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and...
28mC2TACO: Lifting Tensor Code to TACO
15mACCL+: an FPGA-Based Collective Engine for Distributed Applications
41mDesigning software with complex configurations
01h 06mAutomated design space exploration and generation of AI accelerators
19mGuided Tensor Lifting
15mGALA: A High Performance Graph Neural Network Acceleration LAnguage and Compiler