Efficient AutoML with Ludwig, Ray, and Nodeless Kubernetes
Talkat Kubernetes AI Day Europe 2022 · on Monday, 2022-05-16· at 10:05· Pavilion 3, Room D | Level 2 | Central Forum·
summary
Ludwig AutoML uses Ray Tune and heuristics derived from 12 training datasets to pick architectures (TabNet for tabular) and narrow hyperparameter ranges, finding competitive tabular models within one hour and matching manual results on nine unseen datasets within two hours; it adds transfer learning for tabular and uses async hyperband to stop unpromising trials. Running Ray on EC2 (three T4 nodes) cost $81.63 and took 22.6 hours; with Nodeless Kubernetes (Luna) elapsed time drops to 8.75 hours (−61%) and cost drops ~54%; a CPU-only head reduces idle cost with ~15 minutes extra run time.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:53Efficiency of Automl
- 2:56Declarative Machine Learning Systems
- 4:26Ludwig Architecture
- 5:39Scalability
- 6:16Hyper Parameter Search
- 9:12Nodeless Kubernetes Luna Functionality
- 10:41Automl Heuristics for Tabular Data Sets
- 20:51Text Classification
- 22:40Limit the Number of Trials Needed
- 27:22Where Can People Get More Information about Luna
related talks
28mEfficient Deep Learning Training with Ludwig AutoML, Ray, and Nodeless Kubernetes
21mEnd-to-End AutoML with Ludwig on Ray
28mRay on Public-Cloud k8s - experiments, lessons & best practices
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
28mScaling Multimodal Data and Reinforcement Learning with Ray
23mAn Open Source Post-Training Stack: Kubernetes + Ray + PyTorch + vLLM
29mScaling ML Pipelines for Autonomous Driving with Ray
35mReddit's ML Evolution: Scaling with Ray and KubeRay