Building and Managing a Centralized ML Platform with Kubeflow at...
summary
Centralized Kubeflow on Kubernetes integrates CERN SSO (Keycloak), Argo CD, kustomize, TFJob and KNative to offer per-user namespaces with fixed quotas, EOS/HDFS/CVMFS storage mounts and bursting to public cloud. Distributed 3D GAN training uses TensorFlow multi‑worker mirrored strategy locally and on 128 preemptible VMs on GCP with TFJob YAMLs, storing models to S3 buckets. Inference autoscaling increases predictor pods under concurrent load and reported reduced training from 60 hours to 30 minutes and epoch time from one hour to 30 seconds.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:16Introductions
- 0:47What is CERN
- 2:38Motivation for our service
- 3:26Reconstruction
- 4:23Simulations
- 5:23Goals
- 6:19Platform
- 7:06Cluster Layout
- 8:56Deployment
- 9:48Integrations
- 11:35Issues
- 13:00Burst to Public Clouds
- 16:33Automating Distributed Training
- 17:28Service Dashboard
- 18:05Demo
- 21:55Submitting jobs
- 28:35Results
- 29:02Closing remarks
related talks
29mA Scalable Platform for Training and Inference Using Kubeflow
38mBuilding and Managing a Centralized Kubeflow Platform at Spotify
51mMachine Learning using Kubeflow and Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes
14mKubeflow: Where ML Meets Kubernetes
12mKeynote: Building Enterprise AI/ML Platform on Kubernetes
31mKubernetes for Machine Learning: Managing and Scaling AI Workloads
39mDeploy your ML Workloads on Kubernetes with Kubeflow