Gentle introduction to scaling up ML service with Kubernetes + Mlflow
summary
Deploys a Flask + Gunicorn model inside Docker, moves runtime orchestration to Kubernetes using k3s, and adds serverless scaling with Knative so pods scale from zero. Models live in a centralized Mlflow model registry that containers fetch at startup. Demo runs on a three-node cluster (one control plane, two workers, small VMs) for about $30/month on Digital Ocean; setup uses Docker images, Knative autoscaling, and Mlflow and the demo configuration works for lightweight production-like use.
AI-written summary · about
chapters (21)
- 0:00Scaling ML with Kubernetes + MLflow
- 5:28Single container problems
- 7:18Docker and Kubernetes
- 8:02Kubernetes cluster basics
- 9:31Scalability with Deployments
- 12:00K8s Deployments with YAML
- 13:47ClusterIP for internal routing
- 16:20Exposing services outside Kubernetes with NodePort
- 17:49Load Balancer services
- 19:42Serverless ML with KNative
- 22:33ML model binary challenges
- 24:59MLflow Model Registry
- 27:10Registering models in MLflow
- 28:14Final ML service config
- 30:01Lightweight K3s cluster
- 31:47Q&A: Production K8s challenges
- 33:10Q&A: KNative cold starts
- 34:55Q&A: Production K8s configs
- 35:38Q&A: Load balancer subnets
- 36:41Q&A: MLflow multi-tenancy
- 37:48Q&A: K8s vs cloud services
related talks
30mEnd-to-End MLOps with MLflow and Kubeflow
21mMastering the DevOps Machine Learning Pipeline
01h 02mWorkshop: GenOps: Building a MLOps Platform to Support GenAI Workloads with Open...
57mManaging Machine Learning Lifecycle with MLflow
31mMLflow: Platform for Complete Machine Learning Lifecycle
33mOperationalizing Model Lifecycle with MLFlow
23mImproving Machine Learning Workflow
42mDeploying ML models and all the things that go wrong