Reduce, Reuse, Recycle: Is this Mantra under attack with Proliferation of Machine(…)
summary
Machine learning model sizes now grow faster than GPU HBM capacity, causing heavy CPU↔GPU shuttling and allocator fragmentation; experiments show UVM convenience can incur ~1000x slowdown under 4–5x oversubscription for LLMs, while KV‑cache offloading raises latency (≈2.5× with higher request rates). Evaluated fixes include parameter remapping with async transfer (beats VLM/PI and can double throughput), DeepSpeed offload to NVM to fit larger models on one node, and many tradeoffs across HBM, DRAM, NVMe, chiplets, CXL, PIMs and heterogeneous memories.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:49Speaker introduction
- 3:39Memory scarcity in AI
- 5:07Reduce reuse recycle in memory
- 6:33Reducing memory in AI workloads
- 8:00GPU memory capacity gap
- 10:56Reusing memory in parallel systems
- 13:04Recycling memory locations
- 15:33Heterogeneous memory complexity
- 18:023D integration and chiplets
- 20:54Memory allocation complexity
- 23:04CPU-GPU data transfer
- 27:05Unified virtual memory tradeoffs
- 31:03UVM impact in LLMs
- 33:13KV cache management
- 36:54KV cache offloading
- 39:21Dynamic parameter remapping
- 42:37DeepSpeed and NVM offloading
- 45:10Workload-specific memory
- 46:32CXL and PIM innovations
- 47:36Future memory research
- 49:01Conclusion
- 49:22Q&A
related talks
17mMaximizing Utilization for LLM Accelerators
26mEfficiently serving LLMs at scale.
15mUSHER: Holistic Interference Avoidance for Resource Optimized ML Inference
03mAccelerate with purpose
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
35mUtilization Is the Key to Efficiency: What It Takes to Run...
41mBeyond Uptime: Revolutionizing Fintech Reliability