Portable High‑Performance LLM Serving: A Triton Backend for...
summary
Introduces a Triton-only attention backend for vLLM that replaces hand‑written CUDA/HIP kernels and achieves cross‑vendor performance portability by autotuning block/tiling, Q‑block grouping, 2D/3D and persistent launch kernels, and fused online softmax+matmul. Benchmarks on ShareGPT and vLLM latency tests show performance within ~10% of flash attention v4 while the implementation is compact at ~800 lines of Triton code and runs on H100 and MI300 across multiple GPU platforms.
AI-written summary · about
chapters (7)
- 0:00Portable LLM serving intro
- 1:25Performance portability matters
- 3:25Triton for portability
- 5:56Triton attention kernel
- 16:43NVIDIA and AMD GPU results
- 20:39ShareGPT benchmarks
- 23:20Conclusion
related talks
22mEnabling VLLM V1 on AMD GPUs With Triton
31mHow IBM Research Achieved vLLM Platform Portability with Triton Autotuning
20mUnlocking Performance: Harnessing LLMs To Streamline GPU Kernel Development in...
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching
34mBridging the Gap: Compiling and Optimizing Triton Kernels Onto RISC-V Targets Based on MLIR
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
10mMaximizing Kernel Development Productivity Under Performance Constraints