Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service
summary
Model-Switching dynamically swaps DNNs and per-replica thread/replica settings to maximize effective accuracy (baseline accuracy × deadline meet rate) under fixed hardware and fluctuating load. Implemented inside Clipper with batch size 1 and a 1s sampling controller, it profiles and switches among ResNet variants (e.g., ResNet-162, ResNet-101, ResNet-152, ResNet-80, ResNet-34) using a model-based load generator on separate VMs. Results show higher effective accuracy than any single model and avoid costly autoscaling.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:32Deep-Learning Models are Pervasive
- 0:57Computations in Deep Learning
- 2:09Machine Learning Lifecycle (Typical)
- 3:34MLaaS: Challenges and Limitations Maintain QoS under dynamic workloads.
- 5:37Opportunity: DNN Model Diversity
- 6:42Questions in this study
- 7:57What Do Users Care About?
- 10:48Characterizing DNN Parallelism
- 11:58Online Model Switching Framework Model that exhibits best effective accuracy is a function of load
- 12:59Experimental Setup
- 13:42Evaluation: Automatic Model-Switching
- 14:44Evaluation: Effective Accuracy
- 15:51Evaluation: Tail Latency
related talks
11mScaling Distributed Machine Learning with In-Network Aggregation
16mSuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
14mModelKeeper: Accelerating DNN Training via Automated Training Warmup
44mScaling ML from 0 to millions of users
12mMistify: Automating DNN Model Porting for On-Device Inference at the Edge
39mDuplicating Black Box Machine Learning Models
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
12mPreview: ML Serving