Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
Hetu v2 implements HSPMD, an extension of SPMD that adds a top-tier subgroup sharding and asymmetric communication primitives so tensors can be unevenly partitioned and devices run device-specific logic. It provides graph specialization to emit per-device executable graphs and graph switching to reconfigure online. Evaluations on heterogeneous hardware, unstable nodes, and mixed-length data (including a 31‑GPU asymmetric example) match or outperform DeepSpeed and Megatron and reduce reconfiguration time; code and examples are on the Hetu GitHub.
AI-written summary · about
chapters (9)
- 0:00Hetu v2 introduction
- 1:25SPMD limitations in training
- 3:33Asymmetric execution challenges
- 6:04HSPMD for heterogeneity
- 8:10Heterogeneity characterization
- 9:15Graph abstractions for heterogeneity
- 10:17System workflow scenarios
- 12:25Evaluation vs state-of-the-art
- 14:14Summary and conclusions
related talks
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
45mHigh-Performance and Scalable Middleware for HPC and Deep...
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
19mML training acceleration with heterogeneous resources in Bytedance
20mMetis: Fast Automatic Distributed Training on Heterogeneous GPUs
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
19mComposable Distributed PT2(D)
18mThe Promise of Heterogeneous Computing