Lightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
summary
GPU-Scanner is an open-source Kubernetes extension that adds active health checks (PyTorch benchmarks for baselining) and passive health checks (30–40s probes for TFLOPs, memory, NVLink, SRAM, temperature) and wires metrics into Prometheus/Grafana and OpenTelemetry. It deploys via Helm in about four minutes, supports NVIDIA and AMD, integrates with Node Problem Detector and a Django/React/Postgres control plane, and runs auto remediation that can reboot a node — demonstrated recovering an H100 where 1 of 8 GPUs went “off the bus.”
AI-written summary · about
related talks
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
22mKeep HPC Running - an SRE's Guide to Supporting GPUs on Kubernetes
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
09mLightning Talk: Bringing Multi-Tenancy to GPU-Based Workloads
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
11mLightning Talk: Enabling Hot Restart of Stateful Applications Including GPU-Accelerate...
36mProduction GPU Cluster with K8s for AI and DL Workloads
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…