Keynote: Enabling the AI Revolution: Next-Generation Memory Solutions for AI Computing(…)
summary
Claims a working GDDR6-based processing‑in‑memory chip called AiM that puts a lightweight GEMM accelerator inside each bank, with 0.5 TB/s per chip and 1 GB per chip; a prototype AMX card mounts 16 chips to give 16 GB per AMX card and about 8 TB/s per card, targeting LLM response-stage bottlenecks. Reported outcomes: ~13× faster token latency vs H100 GPUs for memory‑bound response work, under 0.5 s versus ~6 s for a 175B example, and about 17% of GPU energy for the J (matrix‑vector) dominated stage. CXL and larger CMS/stacking options are proposed for scale.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:06AI service trends impact
- 3:58Why Transformers dominate
- 7:58Memory demands of embeddings
- 11:33HBM for AI acceleration
- 18:03GPU limits for chat services
- 25:20Processing-in-memory with AIM
- 33:24Deploying AIM accelerators
- 39:46AIM prototype deployment
- 41:39AI data pipeline
- 44:10CXL interconnect fusion
- 47:02Computational Memory Solutions
- 51:17Data hierarchy architecture
- 54:11Q&A
related talks
24mManipulating Data in an Artificial Intelligence World
25mOptimize productivity by meeting next-level demands of the AI era
22mCurrent State of AI
01h 05mReal In-Memory Processing
01h 11mMachine Learning Acceleration in an Embedded World
22mCXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation...
28mSimplifying AI for Edge Compute with HAMi
27mComputers for Super-Human AI