State of Containers and the Convergence of Hyperscale, BigData and HPC
summary
Argument: containers can bridge HPC, hyperscale and AI by shipping the same image from workstation to cluster; a prototype maps host GPUs and user IDs into containers with a Houdini plugin, runs NVIDIA Docker device hooks, and executes MPI on Slurm to span processes across nodes. Demo shows nvidia-smi inside containers and a dam‑break MPI job; obstacles remain: kernel‑bypass drivers, secure UID/GID mapping, shared 30 PB filesystems and scheduler differences. DGX‑2 cited at ~200 TFLOPS, $400k, 10 kW.
AI-written summary · about
chapters (10)
- 0:00Introduction and speaker intro
- 1:07What is HPC and its applications
- 8:53Brief history of HPC computational power
- 13:12Traditional HPC applications and AI convergence
- 16:03Current HPC container usage challenges
- 23:48Kernel bypassing devices in HPC containers
- 26:39Demo: Houdini plugin for device mapping
- 30:13Future roadmap for HPC and AI in containers
- 32:45Enabling science with containers
- 33:22Q&A on Kubernetes and HPC
related talks
36mAn almost perfect match: cloud native and scientific computing
30mRunning AI Workloads in Containers and Kubernetes
15mTransparent GPU Sharing in Container Clouds for Deep Learning Workloads
35mHow does Docker run machine learning on specialized AI
41mBuilding & Operating a Large-scale HPC AI Cluster on Kubernetes
40mAI Factory - Enabling Scalable Innovation & Intelligent Systems
01h 21mHPC in the cloud
05mHami: Dynamic, Smart, Stable GPU-Sharing Middleware In Kubern...