The Race to the Bottom: Low Latency in the age of the Transformer
summary
Shows a practical CPU-first path to transformer inference: a Rust-based Mighty inference server + ONNX Runtime cluster runs 128 inference cores and demonstrates linear scaling to ~3–4k requests/sec (880 rps on 32 vCPU) with typical latencies of 20–60 ms. Covers concrete optimizations—distillation, quantization, pruning, mixed precision, TensorRT and hardware choices—and gives cost examples (claimed 40€/month yields ~14k QPS in one estimate) and caveats where quantization or Python glue actually hurts performance. Demo has minor hiccups but overall validates the CPU-scaling approach.
AI-written summary · about
chapters (7)
- 0:00Introduction to talk
- 4:58Demo of inference server
- 8:14Latency and throughput analysis
- 11:03Cost and hosting breakdown
- 13:15Model optimization techniques
- 16:48Hardware and software efficiency
- 18:29Q&A session
related talks
18mDeploying ML solutions with low latency in Python
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
14mLightning Talk: Every Millisecond Counts: The Fine-tuning Journey of an Ultra-Eff...
01h 00mFrom Tiny Chips to Server Racks: Deploying AI Everywhere with Rust
01h 16mOptimizing workloads with Transformers and Diffusers
31mLow-Latency Solutions for Resource-Constrained Environments
41mEscaping the Cloud: High-Performance AI in your Browser
32mHow to choose the right instance type for ML inference