Large Scale Distributed Deep Learning with Kubernetes Operators
summary
Kubernetes operators automate large-scale training by mapping distribution strategies to CRDs and runtimes: tf-operator (Parameter Server) manages PS-style jobs with CPU-hosted parameter shards and GPU workers; mpi-operator (MPI Allreduce) runs Horovod-style allreduce using MPI and Open MPI; training code needs only small changes — wrap the optimizer with Horovod distributed optimizer and add a broadcast callback — and is launched via TFJob and MPIJob CRDs. Kubeflow common libraries standardize APIs and reuse controllers so operators interoperate and actually run at scale.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:28Orchestration for Deep Learning
- 4:14Parameter Server
- 4:50Distribution Strategy - Reduce
- 5:46Distribution Strategy - AllReduce
- 8:19Kubernetes Operators
- 13:32TensorFlow 101
- 15:47Mirror Strategy in TensorFlow
- 16:50TensorFlow + Horovod
- 21:43Recall: TFJob vs. MPIJob
- 22:37Shared API and Best Practices
related talks
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
46mMachine Learning with Kubernetes
34mDemocratizing Machine Learning on Kubernetes [I]
20mML Ops design patterns with Kubeflow Pipelines
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...
54mScaling Machine Learning Pipelines in Cloud
31mKubernetes for Machine Learning: Managing and Scaling AI Workloads