Lesson learned from orchestrating large-scale GenAI, ML & data on k8s
summary
Metaflow on Kubernetes is production-ready but requires operational work: the team ports Metaflow on Kubernetes to support MPI-style parallelism, PyTorch and Ray and runs transient Ray clusters via AWS Batch for multi-node training. Common surprises: container images grow from ~2GB to 10–20 GB causing slow pulls, autoscalers struggle with non-interruptible long GPU jobs, AZ egress and image caching bite costs, and public cloud GPU scarcity forces multicluster bursting to specialty providers. The platform using SageMaker Studio, Step Functions → Argo, and Metaflow versioning proves reproducible and usable.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 4:41GenAI and foundational models
- 9:24Kubernetes for ML workloads
- 12:36Workload diversity challenges
- 14:44Container image sizes
- 17:15Multi-node GPU workloads
- 18:41GPU scarcity and multi-cloud
- 20:09Autodesk’s ML platform
- 23:44Metaflow integration benefits
- 26:57Ray integration with Metaflow
- 28:25Q&A session
related talks
26mKubernetes and MLOps for Scalable and Reproducible Generative AI
49mKubernetes and MLOps for Scalable and Reproducible Generative AI
35mBuilding a Medical AI with Kubernetes and Kubeflow
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes
51mMachine Learning using Kubeflow and Kubernetes
54mScaling Machine Learning Pipelines in Cloud
28mMachine Learning on Kubernetes at Shell: A Kubeflow Journey