Optimizing GenAI with Intel Gaudi Accelerators on
summary
Ray + Intel Gaudi integration delivers concrete results: native Ray Core/Train/Serve and a VM Gaudi project are available in container images (software v1.7, v1.18 imminent) and use Optima Habana, HCCL and the adapt_transformers hook plus HPU graph and MarkStep to keep execution on device. Measured gains: about 1.75x training and ~1.5x inference overall, >2x on very large models (Llama 70B, Falcon 180B), enabled by 16 HBM stacks (128 GB/device) and HPU resource scheduling — the demos run and scale.
AI-written summary · about
chapters (8)
- 0:00Introduction to talk
- 1:09Intel Gaudi accelerator overview
- 4:02Software stack for Gaudi
- 8:08Enabling Gaudi on Ray
- 12:25Training models with Gaudi
- 14:57Serving models with Gaudi
- 18:33VLLM support for Gaudi
- 26:35Q&A session
related talks
17mFrom Agents to Action: Scaling Agentic AI with Ray on Intel Xeon
29mBenchmarking GenAI Like a Pro: Scaling Experiments, Predicting Performance, and Keeping Your Sanity
18mDriving Innovation with AI
22mGenAI Training In Production: Software, Hardware & Network Considerations
15mBuilding Gen-AI experiences using LLM and LVM
10mCharacterizing AI inferencing stacks
17mEnable Generative AI Everywhere with Ubiquitous Hardware and Open Software
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM