Colocating ML Inference and Training with Fast GPU Memory Handover
summary
A new system called SIRIUS colocates latency-sensitive inference and throughput-heavy training on the same GPU by dynamically resizing training mini-batches (using training elasticity) and centrally managing a shared allocator to achieve a fast memory handover. It completes handovers in under 25 ms, improves inference SLO compliance by an average of 57.0% (up to 97.0%) and boosts training throughput by 2.2× (up to 13.7×), integrates with PyTorch, and avoids PCIe swapping and static partition losses.
AI-written summary · about
chapters (13)
- 0:00GPU memory handover intro
- 0:40GPU underutilization in ML
- 2:04Memory sharing challenges
- 3:52Dynamic swapping limits
- 4:34Training elasticity for memory
- 7:26Fast memory handover challenge
- 8:29Instant memory release
- 9:14Shared GPU memory pool
- 9:57Watermark allocation strategy
- 11:00Evaluation vs baselines
- 13:11Sirius performance gains
- 13:53Sirius multi-GPU scalability
- 14:14Conclusion and release
related talks
19mNo GPU Left Behind: Scaling Online LLM Training With...
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
24mManipulating Data in an Artificial Intelligence World
26mEfficiently serving LLMs at scale.
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
25mMoving Inference to Triton Servers
15mUSHER: Holistic Interference Avoidance for Resource Optimized ML Inference
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...