Optimizing CPU LLM Inference in PyTorch: Lessons From VLLM
summary
vLLM on Arm moves from barely running to about sixfold faster by combining system and kernel work: replacing glibc malloc with MiMalloc yields +133% throughput, fixing a hot atomic loop with Arm LSE adds ~+9%, and exposing pre‑packed BF16 weights through oneDNN avoids repacks for +16%. Attention gets custom QKV/BF16 microkernels and a vectorized cubic softmax (page‑attention 4x, overall +12%), int8/+int4 quant kernels add ~+78% and +6% respectively while keeping ~99% BF16 accuracy, and a shared‑memory communicator cuts tensor‑parallel latency by 43%.
AI-written summary · about
chapters (13)
- 0:00CPU LLM inference intro
- 1:04VLLM stack on Arm CPUs
- 2:50Practical CPU LLM deployments
- 3:53VLLM challenges on Arm
- 4:56System bottlenecks
- 6:24Memory alloc with MiMalloc
- 7:06Thread scaling in libgomp
- 9:58Deep stack integration overhead
- 13:09Kernel optimizations for attention
- 15:17Quantization for linear layers
- 17:25Large models across NUMA nodes
- 19:10Tensor parallelism for latency
- 22:22Optimizations summary
related talks
30mOptimizing vLLM for Intel CPUs and XPUs
08mHacks to Make LLM Training Faster
38mOptimizing vLLM Performance through Quantization
20mUnlocking Performance: Harnessing LLMs To Streamline GPU Kernel Development in...
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching
39mAccelerating Open-Source RL and Agentic Inference with vLLM
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management