From spreadsheet scheduling to Kubernetes: building an on-premise ML platform
summary
On-prem Kubernetes replaces a spreadsheet and raises GPU utilization from ~50% to ~85% (sometimes >90%) by adding a fair-queuing plugin (Q scheduler plugin), Argo CD GitOps, Prometheus metrics feeding Grafana/Loki/Mimir, and a Coder-based developer portal for interactive work. Batch jobs run via standard Kubernetes jobs plus Argo Workflows; GPUs use Nvidia GPU Operator; storage is centralized NVMe with Longhorn for local buffering and MinIO long-term storage. Lessons: prefer Talos OS, avoid Kustomize, expose usage feedback to curb over‑requesting.
AI-written summary · about
chapters (11)
- 0:00ML platform journey
- 2:07Spreadsheet scheduling problems
- 5:00Moving to Kubernetes
- 6:47Queuing with Q plugin
- 9:16Q fairness scheduler
- 11:01Interactive development with Coder
- 15:27GitOps with Argo CD
- 17:33Observability with LGTM
- 20:44Kubernetes platform benefits
- 22:27Lessons learned
- 24:32Q&A
related talks
38mBeyond Kubernetes: Adapting To Spe...
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
18mKeynote: Adapting Kubernetes for Machine Learning Workflows
27mHarnessing Kubernetes for Scalable AI/ML Workloads
46mMachine Learning with Kubernetes
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities