Efficient Deep Learning Training with Ludwig AutoML, Ray, and Nodeless Kubernetes
Talkat Kubernetes Batch + HPC Day Europe 2022 · on Tuesday, 2022-05-17· at 16:20· Room 4F | Event Center·
summary
Runs Ludwig AutoML and Ray on Kubernetes with Nodeless K8s (Luna) to provision just‑in‑time GPUs, replacing a fixed three‑node g4dn.4xlarge Ray cluster. Experiments show elapsed time drops from 22.6 to 8.75 hours, 61% elapsed time reduction, workload cost falls 54% compute cost and idle cluster cost falls 66%, with the option to use a cpu‑only Ray head to cut idle to $0.45/hour while matching model accuracy.
AI-written summary · about
chapters (5)
- 0:00Start
- 4:21Declarative Machine Learning Systems
- 5:51Ludwig Architecture
- 11:37Background the Ludwig Automl Heuristics
- 15:31Control Idle Cost
related talks
28mEfficient AutoML with Ludwig, Ray, and Nodeless Kubernetes
21mEnd-to-End AutoML with Ludwig on Ray
28mRay on Public-Cloud k8s - experiments, lessons & best practices
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
23mAn Open Source Post-Training Stack: Kubernetes + Ray + PyTorch + vLLM
28mScaling Multimodal Data and Reinforcement Learning with Ray
28mTraining Large Language Models on Kubernetes
27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...