Scaling and reproducing deep learning on Kubernetes with Polyaxon
summary
Polyaxon runs on Kubernetes using Helm charts and a CLI to build docker images schedule jobs and track experiments with immutable artifacts and outputs It supports notebooks distributed TensorFlow topologies with workers and PS hyperparameter search and TensorBoard integration The demo shows deployment via Helm project+experiment primitives automated builds and a REST API and claims scaling to 13,000 experiments in less than two hours while keeping immutable experiment artifacts and offering Helm charts and distributed TensorFlow with workers and PS
AI-written summary · about
chapters (22)
- 0:00Polyaxon introduction
- 1:28Speaker background
- 3:55Polyaxon objectives
- 9:36Experimentation workflow
- 11:45Installing Polyaxon
- 15:48Projects and experiments
- 18:56Running experiments
- 23:09Jobs for data management
- 26:20Hyperparameter tuning
- 31:51Grid vs random search
- 35:02Hyperopt optimization
- 41:51Distributed TensorFlow
- 46:31Large-scale experiments
- 52:28Restarting experiments
- 56:48Experiment tagging
- 58:14Resource management
- 1:01:27Jupyter notebooks
- 1:05:28Q&A Polyaxon
- 1:06:53Q&A tracking data
- 1:10:47Q&A scaling resources
- 1:12:37Q&A notebooks experiments
- 1:15:53Roadmap
related talks
38mAutomating Machine Learning and Deep Learning Workflows
19mEfficient Model Exploring and Continuous Delivery With Polyaxon + Kubeflow
34mModern Machine Learning flow with Quilt and Polyaxon
33mCompliant Data Management and Machine Learning on Kubernetes
27mHarnessing Kubernetes for Scalable AI/ML Workloads
25mScaling Machine Learning Models on Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes
29mLarge Scale Distributed Deep Learning with Kubernetes Operators