Universally Deploy Large-language Models via ML Compilation
summary
Claims a universal LLM deployment path by building an ML compiler stack on Apache TVM called MM compiler that adds Python-first pipelines, first-class symbolic shapes, and cross-level IR so new quant formats like AWQ Q4, FP8 and FP4 get supported in a day. It runs Windows/Linux/Mac/iOS/Android/Steam Deck and tiny ARM boards, compiles to WebAssembly/WebGPU for browser demos, and delivers server low-latency results that hit well over 100 output tokens/sec.
AI-written summary · about
chapters (8)
- 0:00ML compilation for LLMs
- 1:03ML vs traditional compilers
- 2:26Python in ML compilers
- 4:11Cross-level optimization
- 6:14Symbolic shape support
- 7:17Universal LLM deployment
- 8:44Low-latency server optimization
- 9:49Live LLM demo
related talks
32mDeploying & serving optimized ML pipelines using graalvm
29mRunning Large Language Models on Laptops...
32mEmpower WebAssembly ML Over Diverse AI Hardware Accelerators Outside the Browser
17mEmpower ML Framework on RISC-V by Backend Graph Compilers
35mThe State of vLLM
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
13mZML: ML framework for Zig
34mOptimizing AI Inference with ML Compilers & Hardware