Kubernetes Deep Dive: Elevating ML Workload Monitoring to Art
summary
A concrete recipe for making ML on Kubernetes resilient: use an always‑on node problem detector daemonset (configured via ConfigMap regexes against kmsg) to set a node condition and call set‑instance‑health so ASG replaces bad instances, avoiding a one‑node minute failure turning into a 600‑minute stall on a 600‑node job. Validate RDMA/EFA with vendor ncom_test across buffer sizes to catch comms issues before runs. Collect device metrics with neuron monitor (daemonset) and scrape to Prometheus/Grafana or CloudWatch via the EKS observability add‑on.
AI-written summary · about
chapters (8)
- 0:00Kubernetes for ML workloads
- 1:07Observability challenges with AI chips
- 2:34Hardware error detection with NPD
- 10:26Pre-job networking validation
- 12:53Monitoring AI chip metrics
- 16:26Prometheus and Grafana demo
- 20:22CloudWatch integration demo
- 23:34ML observability recap
related talks
46mMachine Learning with Kubernetes
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
29mKubernetes Workload Resiliency in Action: Beyond Basics
42mKubernetes for Machine Learning: Managing and Scaling AI Workloads
30mPragmatic SRE for Kubernetes in the Cloud
14mBuilding a high-performance analytics ecosystem on Amazon EKS
06mKubernetes Love Machine Learning, Even on Private Cloud