DISTAL: The Distributed Tensor Algebra Compiler

on YouTube ↗19m

" summary

DISTAL is a compiler that takes tensor index notation plus a tensor distribution notation and a scheduling language with distribute/communicate/rotate to emit distributed C++/CUDA across workstations and supercomputers (evaluated on Lassen up to 256 nodes). It expresses 50 years of matrix‑multiply algorithms (Cannon to COSMA), matches COSMA at ~95% on CPUs and ~85% on GPUs, yields 1.8–3.7x (and a 45x outlier) over Cyclops for higher‑order kernels, and gets 3–4x multi‑GPU bandwidth and up to 10x compute speedups in practice.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 0:06Tensor Computations Are Ubiquitous
3. 0:37Modern Machines Are Distributed and Parallel
4. 1:14Distributed Tensor Algebra Libraries Must Support
5. 2:01SOTA Trades Off Between Performance and Generality
6. 6:05DISTAL's Language is Expressive
7. 6:48DISTAL Competes With Hand-Tuned Code
8. 7:35DISTAL Generalizes High Performance
9. 13:19Conclusion

" related talks

• 14mCompressed and Parallelized Structured Tensor AlgebraMahdi Ghorbani, Emilien Bauer, Tobias Grosser, Amir Shaikhha · OOPSLA 2025
• 32mCompilation of Dynamic Sparse Tensor AlgebraSaman Amarasinghe, Stephen Chou · OOPSLA 2022
• 28mFunctional Sparse Tensor CompilationAmir Shaikhha, Shideh Hashemian · FProPer 2024
• 18m(De/Re)-Composition of Data-Parallel Computations via Multi-Dimensional(…)Ari Rasch · PLDI 2024
• 20mCompiling Structured Tensor AlgebraMahdi Ghorbani, Amir Shaikhha, Mathieu Huot, Shideh Hashemian · OOPSLA 2023
• 17mDistributed Sparse Computing with Legate SparseRohan Yadav · Sparse 2025
• 11mQiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a...Qi Guo, Xuehai Zhou, Shouyang Dong, Yuanbo Wen, Jun Bi, Di Huang, Jiaming Guo, Jianxing Xu, Ruibai Xu, Xinkai Song, Yifan Hao, Ling Li, Tianshi Chen, Yunji Chen · OSDI '25
• 18mCompiling Strassen-like Matrix Multiplication Algorithms to Fast CUDA KernelsAbhinav Jangda · PLDI 2026