Lightning Talk: My Job Says 'Running' but Nothing's Running: Kubernetes Status Reality...
summary
Kubernetes often reports a distributed PyTorch/MPI job as Running while no training progresses because Kubernetes only validates per-pod readiness probes and scheduling, not inter-pod network connectivity or role-level progress. Check DNS, network policies, init-container completion, and PCIe/NCCL GPU communication before starting; instrument and monitor GPU utilization, checkpoint file timestamps and MPI ranks during runtime; implement pre-flight connectivity and GPU-comm tests and check GPU utilization and checkpoint timestamps as liveness/aggregation checks to avoid wasted GPU hours.
AI-written summary · about
related talks
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
05mLightning Talk: Kubernetes Jobs and the Sidecar Problem
38mKubernetes for AI Workloads – What Works and What… Doesn’t
06mLightning Talk: Is Your Kubernetes Cluster's DNS Working?
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
46mMachine Learning with Kubernetes
09mLightning Talk: Optimize the Ray Schedule and Autoscaling on Kubernetes Cloud: A Hetero...