Beyond the DC Walls: Building a Nationwide GPU Fabric With KubeVirt and Wide-Area…
summary
Nationwide GPU fabric stitches eight data centers across ~2,000 km over a 100 Gbit wide‑area L2 IOWN APN, runs Kubernetes with KubeVirt and SKT’s Petabyte AI Cloud, and exposes NVLink, VFIO passthrough, SR‑IOV, and GPU‑direct RDMA inside VMs. Provisioning drops from five business days to minutes, Tokyo–Osaka RTT is ~9 ms, GPU‑to‑GPU all‑reduce hits ~12.9 GB/s and storage writes/read ~11.3/10.1 GB/s. Current workaround uses a per‑device “jammer board”; DRA-based device allocation replaces it.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:25Why build nationwide GPU fabric
- 3:15Introducing Goat testbed
- 4:37Core requirements for testbed
- 5:44Why Petasense Air Cloud fits
- 8:15Goat architecture overview
- 9:00Design goals for wide-area L2 testbed
- 10:05NVLink virtualization challenges
- 12:34Enabling GPU Direct RDMA
- 14:02Preserving PCIe/NUMA in VMs
- 15:45Full system architecture
- 16:07Performance Tokyo vs Osaka
- 17:33Why tuning is critical
- 18:17PCIe topology alignment
- 19:00Current workaround: Jammer board
- 20:25Dynamic Resource Allocation
- 22:33How DRA works in Kubernetes
- 24:18DRA scheduling process
- 25:42DRA with VM creation
- 26:46Future: shared GPU clusters
- 27:51Petasense Cloud transition
- 28:12Next milestones: rack-scale
- 28:54Key takeaways
related talks
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
08mContainerization – It’s Not Just for CPUs Anymore: Introducin...
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
25mHigh Performance Networking for Distributed DL Training in Production K8s
14mWhy We Don’t Need More Data Centers
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
31mBut we have GPUs at home - on-prem ai with kubernetes, Openstack & KubeVirt