Memory Harvesting in Multi-GPU Systems with Hierarchical Unified Virtual Memory
summary
Implements a hierarchical unified virtual memory (HUVM) that spills GPU pages to neighbor GPUs over NVLink instead of host PCIe, and ships a centralized memory manager memHarvester that uses pre-eviction, parallel fetch and multi‑pass prefetch. Built and evaluated on AWS 4‑GPU NVLink nodes with graph analytics and PyTorch DNNs; harvesting as little as 5% spare neighbor memory yields large gains—microcases show ~3.6× speedup and overall experiments report up to 2.71× improvement versus stock UVM with modest neighbor interference.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:29Performance overhead for memory oversubscription
- 5:24Data-path: Hierarchical Unified Virtual Memory
- 7:111. Hiding eviction latency
- 9:222. Reducing fetch latency
- 13:47Evaluation
- 14:05Workload running scenarios
- 14:49Case-A (1 harvests 3)
- 17:07Individual technique performance breakdown
- 19:09Conclusion
related talks
14mServing Frontier MoE Models at the Lowest Token Cost
20mPassthrough/Headless GPU Gets Ahead
01h 00mMemory Resources in a Heterogeneous World
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
15mNixie: Efficient, Transparent Temporal Multiplexing for Consumer GPUs
33mGPU Para-Virtualization on Xen
20mSpeed Up Creation of a VM With Pass Through GPU
04mLightning Talk: Dynamic GPU Fraction and Sharing With Cloud Native Principle