Street fighting techniques for multi tenant machine learning...
summary
Deploys a MapR CSI storage cluster inside Kubernetes, installs Kubeflow and JupyterHub, and launches training and Spark jobs so the demo actually runs end-to-end. The presenter provisions 50 nodes in 4m14s, shows a GC cluster with ~800 vCPUs and 5,200 GB RAM, and launches a TensorFlow job with 40 workers via a Kubeflow operator; pods, persistent volumes and operators all come up and the TensorFlow job appears in the Kubeflow dashboard as running.
AI-written summary · about
chapters (25)
- 0:00Introduction to talk
- 1:04Why this talk matters
- 2:52Early struggles in ML environments
- 4:36Current state of ML on Kubernetes
- 6:24Setting up storage in Kubernetes
- 9:56Installing Kubeflow components
- 10:58Stateful vs stateless in ML
- 13:11Pods as building blocks
- 14:56Resource isolation with namespaces
- 17:09Persistent storage for ML
- 18:34Challenges of copying data
- 21:07Launching Jupyter notebooks
- 21:55Data science workflow patterns
- 22:58Understanding Kubernetes operators
- 24:02Job operators for training
- 26:36Running jobs in user namespaces
- 27:38Smart pod scheduling techniques
- 28:44Avoiding hand-rolled containers
- 30:53Separating compute from storage
- 31:57Building custom operators
- 33:00Exploration with notebooks
- 34:01Monitoring training jobs
- 34:44Serving trained models
- 35:47Handling YARN workloads
- 37:15Summary and Q&A
related talks
54mMachine learning at scale using Kubeflow with Amazon EKS & Amazon EFS
51mMachine Learning using Kubeflow and Kubernetes
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
46mMachine Learning with Kubernetes
31mKubeflow Intro – Michał Jastrzębski & Ala Raddaoui, Intel
46mDeploying scalable Machine Learning models with Kubernetes
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...
35mKubeflow: Data Scientist ready?