Lightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
Lightningat Cloud Native + Kubernetes AI Day Europe 2025 · on Tuesday, 2025-04-01· at 17:00· Level 1 | Hall Entrance N10 | Room G·
summary
A Kubernetes-native benchmark for distributed ML training that automates environment control (PVC, daemonset kernel/config tweaks), launches MPI jobs and collects runtime and postmortem metrics to find real cluster issues — e.g. firmware updates causing ~10% throughput drops and single GPUs dragging a 1,000‑GPU fleet by ~5%. Built as a lightweight operator using Argo Workflows, MPI Operator, and telemetry via Prometheus + Grafana + DCGM, with Torch Profiler/TensorBoard integration; currently runs Megatron and GPT workloads and is partly open source and usable.
AI-written summary · about
chapters (7)
- 0:00Distributed ML benchmarking intro
- 1:27Distributed ML platforms background
- 3:16GPU cluster challenges
- 5:28Controlled benchmarking need
- 6:32Existing tools evaluation
- 8:47New tool design
- 10:58Current status and future
related talks
46mMachine Learning with Kubernetes
28mTraining Large Language Models on Kubernetes
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
51mMachine Learning using Kubeflow and Kubernetes
23mBenchmarking Machine Learning Workloads on Kubeflow