From Tiny Chips to Server Racks: Deploying AI Everywhere with Rust
summary
Tests every common inference engine (ONNX Runtime, llama.cpp alternatives, executort, etc.), benchmarks Rust frameworks (ort/ort-rs, burn, candle, tract), and reports concrete results: a handbuilt Rust engine reduces ROM from ~18 MB to 0.75 MB, RAM from ~30 MB to ~6 MB and cuts latency to 8 ms worst-case versus ~16 ms on ONNX Runtime. Notes which formats matter (GGUF), which backends exist, that cross-compilation via cargo build works well, and that DSP/NPU vendor APIs are still missing.
AI-written summary · about
chapters (24)
- 0:00Deploying AI with Rust
- 2:33Speaker background
- 8:02AI model deployment challenge
- 9:26AI training and inference frameworks
- 15:08Model formats for inference
- 18:23Choosing inference engine
- 20:11Rust for AI inference
- 21:17Rust ML frameworks
- 27:41ort crate for inference
- 28:43Burn framework
- 35:26Candle with GGUF
- 38:17Tracked CPU engine
- 41:30Other inference frameworks
- 43:14AI Acoustics requirements
- 44:18CPU acceleration with NDArray
- 46:50GPU programming in Rust
- 51:33DSP and NPU challenges
- 52:15Custom inference engine
- 53:42Rust for AI everywhere
- 54:46Conclusion and future
- 55:06Q&A: Hardware choices
- 56:53Q&A: Cross-compiling Rust
- 58:03Q&A: VLLM and SG Lang
- 58:46Q&A: Accessible inference
related talks
29mRust for ML Infrastructure: A Case Study in Building an ONNX Runtime
19mEnhancing Gen-AI with Rust: Scalable AI Systems
20mMastering Rust on Edge and Cloud — AI/IoT Case Studies
23mThe Race to the Bottom: Low Latency in the age of the Transformer
25m"Rust for AI & Accelerated Computing"
38mTokio, Rayon, and the GPU: Profiling a Radiology AI Pipeline in Rust
26mAllocating Less: Really Thin Rust Cloud Apps
56mWhen to Choose Rust