Efficient Training of Video Generation Foundation Model at ByteDance
summary
Covers ByteDance’s methods for scaling training of large video-generation models on thousands of GPUs using PyTorch. Describes use of PyTorch Compile for kernel fusion, checkpointing and CPU/GPU offloading for memory-limited large-model training, a combination of Sequence Parallelism and Hybrid Sharding Data Parallelism for near-linear distributed scalability, and a dynamic load‑balancing scheme to handle variable‑length videos.
AI-written summary · about
related talks
19mML training acceleration with heterogeneous resources in Bytedance
24mTraining large scale models using PyTorch
28mBuild, Buy and the Golden Ratio: How Theator scaled up its continuous training framework
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
25mInto Generative AI with PyTorch Lightning 2.0
25mDistributed Training in PyTorch: Zero to Hero
22mMovie Gen Video: State-of-the-art Video Generation