Training Foundation Model Workloads on Kubernetes at Scale W...
summary
Vela runs on Kubernetes with about 1,000 A100 GPUs and aims for near‑zero idle capacity; MCAD queues AppWrapper CRDs and dispatches heterogeneous frameworks (PyTorch, Ray, Spark, TensorFlow) with zero code changes on target operators. It enforces gang scheduling, priority/preemption, hierarchical quota borrowing, and automatic requeueing for node/GPU failures. InstaScale autoscaler acquires whole gangs of resources, reuses prior allocations, scales to zero, and ties dispatch to actual cluster capacity to raise utilization.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:46Foundation models hardware
- 3:34AI training platform needs
- 6:24Open Data Hub stack
- 7:50Codeflare SDK intro
- 8:53Dispatching vs scheduling
- 9:36MCAD features
- 11:41Fault tolerance in MCAD
- 12:46Hierarchical quota in MCAD
- 14:58Bring your own scheduler capability
- 15:41MCAD dashboard
- 16:02InstaScale autoscaler
- 17:54MCAD & InstaScale recap
- 20:02MCAD multicluster dispatcher
- 20:44Roadmap & future
- 22:30Community & next steps
- 23:33Q&A on ML resources
related talks
27mHarnessing Kubernetes for Scalable AI/ML Workloads
01h 22mWorkshop: Optimizing Kubernetes Cluster Scaling for Adv...
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…