Scaling ML to the next level: Hosting thousands of models on SageMaker
summary
Explains how Amazon SageMaker multi-model endpoints host thousands of models from S3 on a single endpoint, lazy-loading models on first invoke and evicting by LRU; shows a notebook demo deploying four XGBoost city models, autoscaling from 1 to 4 instances on >10 invocations/instance/min, and warm vs cold latency differences; lists tuning knobs (OMP, inter/intra op, MMS workers, MKL) and reports optimized throughput jump to 44,000 invocations per minute while serving thousands of models at lower cost.
AI-written summary · about
chapters (6)
- 0:00Introduction to MME
- 2:54How MME works
- 4:25MME demo and auto scaling
- 11:35Optimization lessons
- 17:01Zendesk customer story
- 26:33Conclusion
related talks
44mScaling ML from 0 to millions of users
39mTrain ML models at scale with Amazon SageMaker Training
26mManaging ML Models @ Scale
20mIndustrial-Scale Machine Learning
31mTrain and tune ML models to the highest accuracy using Amazon SageMaker
41mDeploy ML models for inference at high performance & low cost
52mGenerate ML predictions without writing any code
01h 00mHigh-performance & cost-effective model deployment with Amazon SageMaker