Scaling Machine Learning jobs with Kubernetes
summary
Textkernel runs TensorFlow training on a fixed on‑prem Kubernetes cluster with Docker images built by Jenkins from GitLab, using pod anti‑affinity so each training job occupies one node; doubling CPUs roughly halves epoch time. A 13‑engineer ML team supports ~18 languages with ~6 models per language, so hundreds of jobs; hyperparameter sweeps are split into separate Kubernetes jobs via a small custom scheduler using the Kubernetes Python API to run trials in parallel. Kubeflow and distributed TensorFlow are evaluated but weren’t the initial solution.
AI-written summary · about
chapters (10)
- 0:00Scaling ML jobs
- 1:25What TechScreen does
- 3:09Separating modeling from ops
- 4:57Manual job scheduling problems
- 7:43Why Kubernetes fits
- 11:56Automating with Docker/K8s
- 16:11Scheduling training jobs
- 17:57Scaling with hyperparameter tuning
- 20:52Automation and scalability recap
- 21:56Q&A
related talks
46mMachine Learning with Kubernetes
31mKubernetes for Machine Learning: Managing and Scaling AI Workloads
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
51mMachine Learning using Kubeflow and Kubernetes
54mScaling Machine Learning Pipelines in Cloud
22mThe Containerization of Machine Learning: TensorFlow: Kubernetes, and Kubeflow
26mMachine Learning Meets K8s
28mProductionizing Machine Learning Models at Scale with Kubernetes