AI in Kubernetes: How to Get Started?
summary
Shows a concrete path: using Flyte to run a Python ML pipeline that fine‑tunes a foundation model, applies quantization/LoRA-style compression, and serves with a model server on GPU nodes (demo requests a Tesla T4 and completes all steps). Points out hard limits — Llama 3.1 (405B) needs >800 GB GPU memory per node — and operational gaps: gang-scheduling and cluster-autoscaler integration are missing, so teams often use one huge pod per node (OpenAI scale ~1,300–1,500 nodes). Demo pipeline runs end‑to‑end.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:08What is AI
- 4:42AI in Kubernetes
- 6:07AI pipeline in K8s
- 8:55Hardware demands of large language models
- 10:24Distributed AI training challenges
- 12:12Gang scheduling limits
- 13:38Distributed AI serving
- 16:54OpenAI’s K8s AI at scale
- 17:58Why Kubernetes is a good fit for AI
- 18:41AI platform on K8s
- 20:26Flyte for AI workflows
- 23:15Flyte demo
- 32:10Summary and challenges
- 33:02Q&A
related talks
27mHarnessing Kubernetes for Scalable AI/ML Workloads
16mAI + Kubernetes: What Beginners...
38mKubernetes for AI Workloads – What Works and What… Doesn’t
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes
28mScaling AI Infrastructure
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
17mDevOps for Private AI: A Scalable Platform Using Kubeflow