Cocktail: A Multidimensional Optimization for Model Serving in Cloud
Talkat 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22) · 2022-04-04..04-06·
summary
Cocktail is an ensemble-based model-serving framework that dynamically selects a small set of models per request using a class-wise accuracy dictionary and weighted voting, while an adaptive RM layer does distributed proactive autoscaling with importance sampling and uses AWS EC2 spot instances for cheaper capacity. A prototype on AWS EC2 shows 1.45x lower deployment cost, about 2x lower latency, and meeting target accuracy for up to 96% of requests by typically serving five to six models instead of ten.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:31MODEL SERVING HOSTED ON CLOUD
- 1:52MODEL SERVING CHALLENGES
- 3:07PRIOR WORK IN MODEL SERVI
- 4:00Model Ensembling Framework
- 4:45MODEL SPACE EXPLORATI
- 5:21FULL ENSEMBLE
- 6:18FULL ENSEMBLING COST
- 7:00WHAT CAN WE DO?
- 7:52DYNAMIC MODEL SELECTION
- 8:33COCKTAIL- MULTIDIMENSIONAL OPTIMIZATION FOR ENSEMBLE LEARNING IN CLOUD
- 9:01OBJECTIVE FUNCTIONS
- 11:11EVALUATION AND SETUP
- 11:38MAJOR RESULTS
- 13:06SUMMARY
related talks
45mModel serving: which framework to choose?
37mVibe Computing: AI-Powered Multi-Cloud Orchestration With Cloud-Barista
31mServing Machine Learning Models at Scale Using KServe
25mModel as a Service for Real-Time Decisioning
29mProactive and Polymorphic Adaptation of Multi-cloud Deployments
20mMArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving
14mCocktailer: Analyzing and Optimizing Dynamic Control Flow in Deep Learning
27mScaling ML to the next level: Hosting thousands of models on SageMaker