Running ML Challenges on Kubernetes (Part 2)
summary
CERN runs a production prototype that turns Codabench into a Kubernetes-native challenge platform: a Helm chart is merged upstream and the executor now creates executor and scorer pods via the Kubernetes API, so submissions are scheduled and GPU resources are managed properly. The platform integrates with Kubeflow notebooks and adds a JupyterLab extension to submit code directly to Codabench, supports S3 for dataset storage, enforces submission limits and multi‑phase scoring, and handles terabyte‑scale physics datasets while work continues on PV mounting and per‑user namespaces.
AI-written summary · about
chapters (7)
- 0:00CERN and ML challenges
- 2:28Gamification concept
- 6:23Codabench architecture setup
- 9:33Kubernetes with Codabench
- 10:58Managing competitions
- 14:48Notebooks with Kubeflow
- 17:38Future improvements
related talks
29mStreamlining Competitive Data Science at CERN: Running ML...
46mMachine Learning with Kubernetes
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
24mA Better and More Efficient ML Experience for CERN Users
33mCompliant Data Management and Machine Learning on Kubernetes
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...
37mA Day in the Life of a Data Scientist. Conquer ML Lifecycle on Kubernetes