Scaling to Millions of ML Models to Solve the Problems of SRE and Security
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Friday, 2020-11-20· at 16:00· Intrado Virtual Event Platform,·
summary
Claims scaling millions of models over petabytes of metrics and logs using Kubernetes, Prometheus, Cortex, Databricks Spark and Apache Arrow with per-node Envoy inference and model distribution. Engineers build an isolated Databricks setup via an ingress/egress gateway called VoltMesh to expose only Prometheus/Cortex/Elasticsearch APIs. Performance details: group-map pandas UDFs + Arrow cut Python serialization overhead up to ~100x; Spark launches ephemeral clusters to parallelize training and reduce costs; the pipeline runs in production and handles millions of time series.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:15Meet the speakers
- 0:58Agenda
- 1:38What is Volterra
- 2:57Machine Learning Applications
- 4:48Metrics and Locks
- 7:18Complex Architecture
- 9:08Infrastructure
- 10:46Scaling
- 11:55Spark and Kubernetes
- 12:33Databricks Integration
- 21:43Apache Arrow
- 22:49Group pandas API
- 23:40Code snippet
- 26:37Conclusion
related talks
33mDealing with the versioning of production-ready models
31mBeyond One Model: Scaling, Orchestrating & Monitoring
33mDeploying and Scaling Spark ML and Tensorflow AI Models
28mBuilding Developer-Centric ML Inference Platforms
28mEfficient Time-Series Forecasting with Thousands of Local Models on Databricks
43mDeploying Large Spark Models to production #bbuzz
11mScaling R with Spark
29mMachine learning in production