Keep HPC Running - an SRE's Guide to Supporting GPUs on Kubernetes
summary
Shows a working approach that queries DCGM-Exporter metrics (via NVIDIA DCGM and NVML bindings) from Prometheus and implements a custom node-problem-detector plugin that flips a node to NotReady when GPU temperature or memory/compute pressure crosses thresholds. Demonstration on a home lab with single‑GPU nodes proves the scheduler stops placing jobs on unhealthy cards, reducing mid‑job failures. Covers capacity planning, load testing, chaos/tabletop rehearsals, and runbooks to automate recovery and validate SLOs.
AI-written summary · about
chapters (14)
- 0:00HPC and SRE on Kubernetes
- 1:04HPC vs traditional workloads
- 3:12GPU scheduling challenges
- 4:39Upstream data availability
- 6:04Capacity planning for GPUs
- 8:09Observability in Kubernetes
- 9:16GPU health checks with DCGM
- 10:21GPU readiness with Node Problem Detector
- 11:46Load testing HPC clusters
- 12:30Chaos testing and resilience
- 13:52Incident response preparedness
- 14:36Human factors in reliability
- 15:38Future of GPU open standards
- 16:41Q&A
related talks
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
36mProduction GPU Cluster with K8s for AI and DL Workloads
07mYour GPU delivery is here! Now what? A 5 minute guide to make your CTO happy
32mAre You Really Out of GPUs? How to Better Understand Your GPU...
31mGPU Hunter: Architecting Global GPU Availability With MultiKueue
41mBuilding & Operating a Large-scale HPC AI Cluster on Kubernetes