Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
summary
Reports a system called Optimus that cuts end-to-end MLLM training time by ~20.5–21.3% on 3,072 GPUs (ViT-22B encoder + GPT-175B backbone) by turning idle GPU bubbles into useful work. It places encoder and LM states on every GPU via separate parallel plans, uses a greedy bubble-scheduling algorithm with global ordering, and decomposes encoder layers to kernel-level scheduling to fill sub-millisecond (~300 µs) tensor-parallel gaps. Compares vs Megatron, FSDP, APA.
AI-written summary · about
chapters (8)
- 0:00Optimus introduction
- 1:24Multimodal LLM challenges
- 3:31Exploiting GPU idle time
- 5:57Optimus design: encoder & LM states
- 8:27Scheduling complex dependencies
- 10:32Kernel-level scheduling
- 11:37Performance evaluation
- 14:03Summary
related talks
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
23mOperationalizing Large Language Model Training Pipelines
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
16mAccelerating Design Space Exploration for LLM Training Systems with Multi-experiment...
01h 18mOptimizing LLM Training and Inference Performance on GPUs
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
26mEfficiently serving LLMs at scale.
19mNo GPU Left Behind: Scaling Online LLM Training With...