Exploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
summary
Kubernetes-native distributed cache (Alluxio) uses NVMe, RDMA and GPU Direct Storage with UCX and a Netty data plane to feed GPUs. Single-worker FIO reaches ~7–8 GB/s sequential and ~7 GB/s random, and performance scales roughly linearly across workers. Design uses consistent hashing and fully sharded metadata with no master. Real deployments raise GPU utilization from ~30% to ~90% for LLMs, expand fleets 500→2,000+, and cut checkpoint distribution latency 20–30 min to 3–5 min.
AI-written summary · about
chapters (10)
- 0:00Introduction and agenda
- 0:56I/O challenges in AI training
- 5:23Options for efficient I/O
- 8:35Alluxio architecture design
- 13:43Distributed caching internals
- 17:11Benchmark results
- 20:34Network and transport tuning
- 23:43Faster checkpointing via GDS
- 25:27Case study and real-world impact
- 30:32Q&A and conclusion
related talks
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
28mMaximize GPU Utilization for Model Training
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
25mHigh Performance Networking for Distributed DL Training in Production K8s
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
33mScaling KV Caches for LLMs: How LMCache + NIXL Handle Network and Storage...