Enable Generative AI Everywhere with Ubiquitous Hardware and Open Software
summary
Implemented low‑precision (bfloat16/INT8/INT4), Flash/Efficient Attention, optimized KV‑cache access, RoPE kernel fusion, tensor parallelism and AMX/EMX CPU instructions inside Intel Extension for PyTorch and PyTorch. On a CPU server (two 4th‑gen Xeon style chips) LLaMA‑7B averages ~40 ms/token and LLaMA‑13B ~72–76 ms/token. Stable Diffusion fine‑tuning completes in about 5 minutes on a 4‑node RAPID setup and single‑image inference runs ~5 seconds, demonstrating CPU‑only generative AI is practical.
AI-written summary · about
chapters (9)
- 0:00Generative AI everywhere
- 1:08Hardware beyond GPUs for AI
- 2:15Intel’s AI acceleration in CPUs
- 4:44Software ecosystem for generative AI
- 6:48Intel Extension for PyTorch
- 9:46Optimizing LLMs on CPUs
- 11:13LLM inference benchmarks
- 13:26Stable Diffusion speed and tuning
- 15:36Enabling AI with CPU and open software
related talks
15mBuilding Gen-AI experiences using LLM and LVM
15mThis or That with Takara AI
23mThe Race to the Bottom: Low Latency in the age of the Transformer
52mSimplifying the adoption of generative AI for enterprises
16mDe-mystifying PyTorch for ASICs: When (and Why) To Move Your Development To AI A...
22mGenAI Training In Production: Software, Hardware & Network Considerations
14mAccelerating AI inference workloads
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...