USHER: Holistic Interference Avoidance for Resource Optimized ML Inference
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
System claims a concrete 2.6× higher goodput and 3.5× better cost-efficiency on large-scale tests by combining a fast kernel-based estimator that predicts per-model compute/memory from GPU kernels, a heuristic scheduler that picks batch size, replica count and placement to hit latency SLOs while maximizing utilization, and an operator graph merger that fuses weight-overlapping operators to reduce L2 cache churn; implemented on thousands-of-GPUs workloads and evaluated on 20 CNN/LLM models with Azure-like traces.
AI-written summary · about
chapters (6)
- 0:00Motivation and problem scope
- 4:51Usher system overview
- 5:44Resource estimation and scheduling
- 8:36Minimizing cache interference
- 12:11System evaluation setup
- 12:25Results and conclusion
related talks
24mCost and Performance Optimization of LLM Inferencing.
20mOPER: Optimality-Guided Embedding Table Parallelization for Large-scale...
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
19mResource Management Innovation for AI Workloads in Cloud Computing
31mThe Evolution of Multi-GPU Inference in vLLM
19mOffloading-Efficient Sparse AI Systems
11mEngineering Scalable ML Infrastructure for Modern Apps