Automatic Resource Configuration for Kubeflow TrainJobs
summary
Feature adds a simple plugin protocol where an external Kubernetes controller (a recommender) mutates TrainJob resources and Kubeflow Trainer enforces guardrails like timeouts and quota enforcement. The system is opt‑in via an annotation and requires jobs to start in a suspended state so recommenders can patch resources; example recommender raises a failed PyTorch fine‑tune from 1 GPU to 4 GPUs while preserving per‑device batch size. Team benchmarks: ~90,000 jobs run, 12,000 fine‑tuning samples used to train a model with ~91.6% holdout detection.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:55Manual allocation problems
- 9:19Solution with recommender plugins
- 11:49Live demo: auto patching
- 16:40Recommenders as ML models
- 19:08Recommender types
- 22:22Building model with benchmarks
- 24:50Kubernetes/Kubeflow status
- 26:37Call to action
- 28:27Q&A
related talks
29mKubeflow Trainer Observability: Real-Time Progress Tracking for Reliab...
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...
16mStreamline LLM Fine-tuning on Kubernetes With Kubeflow LLM Trainer
29mKubeflow: TensorFlow on Kubernetes
20mEmpowering ML Workloads With Kubeflow: JAX Distributed Training and LL...
46mDeploying scalable Machine Learning models with Kubernetes
35mHyperparameter Tuning Using Kubeflow