Serving Heterogeneous Machine Learning Models on Multi-GPU Servers...
summary
A multi-model scheduling framework implements spatial and temporal packing of GPU resources into configurable virtual partitions called gpulets, profiles throughput per partition to pick cost‑effective sizes, and uses shadow contexts to reorganize partitions with low overhead. The prototype auto-scales up to 8 GPUs, maintains SLO violations under 1% in traces like Game and Traffic, gains 61.7% throughput over time‑sharing, and improves schedules further by 7.5% using an interference prediction model.
AI-written summary · about
chapters (9)
- 0:00Start
- 1:07Approaches for Scheduling Machine Learning Inference Tests on Gpu
- 4:07Spatial Temporal Scheduling
- 5:37Proposed Gpu Led Scheduling Framework
- 8:52Find the Number and Size of Cost-Effective Partitions
- 10:26How Partitions Are Located To Input Rates
- 12:17Dynamically Reorganizing Partitions
- 13:19Evaluation Results
- 16:03Conclusion
related talks
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
42mOne GPU, Four Retrieval Modes: Multi-Model Search Serving
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
36mAccelerating High-Performance Machine Learning at Scale i...
20mScheduler and Sharding Considerations for Network Efficiency
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
15mUSHER: Holistic Interference Avoidance for Resource Optimized ML Inference