Build and Monitor Machine Learning Services in Kubernetes

on YouTube ↗22m

" summary

Recreates a 24,000-clip “clock” project using scraping, manual fallback and a $200,000,000 museum pipeline that required a 60-person team, then shows a solo approach: download via headless browsers, run YOLO object detection, extract snippets with ffmpeg, and serve models in Kubernetes using Kubeflow, Seldon and TensorRT inference servers. Demonstrates local GPU dev with microk8s, NVIDIA containers, observability via distributed traces and daemonsets, and edge deployment onto Jetson Nano while measuring GPU utilization and frames-per-second.

AI-written summary · about

" chapters (26)

1. 0:00Intro
2. 3:56DeepFakes & Nicholas Cage
3. 4:39How to build a deep learning model
4. 4:59Actual model used for a project
5. 6:12Driver updates can speed up your code 300% (?!)
6. 6:32Training Computational Needs Can Be Staggering
7. 7:36Inference Gets Pushed Out to the Edge
8. 8:09Datasets Become Integral Part of Your Code
9. 8:58More Container Adoption
10. 9:11And Containers Usually Means Orchestration
11. 9:55Moving from managing and deploying to individual servers to pushing code to a "sea of infinite computation"
12. 10:07etod, default database to manage kubernetes state, recommends 5 server instances to durability
13. 11:55More Pieces to Manage and Deploy
14. 12:38Kubeflow supports both TensorRT and PyTorch for Inference
15. 14:00NVIDIA Container Registry
16. 14:40Py Torch & FFMPEG w/ Hardware Acceleration Dockerfile
17. 15:05Example YAML Manifest for Kubernetes
18. 16:02For example, pipelines.
19. 17:23Compress Distributed System Complexity with Traces
20. 17:52Individual Trace
21. 18:11See History of Machine State with Metrics
22. 18:47Add Observability with Daemon Sets
23. 19:18See Auditable Trail of Side Effects with Logs
24. 20:20Bin-e, a self sorting recycling bin
25. 20:41Dab & T-Pose controlled lights
26. 21:11Jetson Nano $99 GPU Accelerated Machine

" related talks

• 28mProductionizing Machine Learning Models at Scale with KubernetesLuigi Patruno · TWIMLcon: AI Platforms 2019
• 42mKubernetes for Machine Learning: Managing and Scaling AI WorkloadsDrishti Jain · DevConf.IN 2026
• 46mDeploying scalable Machine Learning models with KubernetesAlessandro Vozza · Future Tech 2019
• 29mTeam Teardown: MLOps in the Cloud for IoTDanielle Dean, Mathew Salvaris, Mohan Mupiddi · TWIMLcon: AI Platforms 2021
• 51mMachine Learning using Kubeflow and KubernetesArun Gupta · Devoxx Belgium 2019
• 46mAccelerating Machine Learning DevOps with KubeflowDerek Ferguson · GOTO Chicago 2019
• 32mGPU Machine Learning From Laptop to CloudMark Puddick · KubeCon + CloudNativeCon Europe 2019
• 29mA Scalable Platform for Training and Inference Using KubeflowDiana Gaponcic, Philipp Gadow · Kubeflow Summit Europe 2024