Lightweight, low-overhead, high-performance: machine learning directly in C++
summary
Claim: rewriting pipelines in C++ yields much smaller and faster deployments. Uses mlpack + Armadillo with low-level BLAS (OpenBLAS) and aggressive statically linked to OpenBLAS builds. Demonstrated a language-detection microservice (softmax on byte n‑grams) that runs instantly, trains to ~96% on the demo set, and serializes to ~176 KB. Measured sizes: PyTorch container ~6.5 GB, slim Python ~50–330 MiB, C++ containers down to single‑digit MiB and a stripped binary ~1.5 MiB.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:08Python ML deployment problems
- 8:50ML stack and abstraction costs
- 13:27C++ deployment advantages
- 18:04Demo: Language detection in C++
- 25:54Container sizes: Python vs C++
- 29:02Optimizing C++ binary size
- 30:49Conclusion and ecosystem tools
- 39:01Q&A
related talks
01h 53mTinyML: Machine Learning’s Future is Tiny and Bright
48mLeveraging the Power of C++ for Efficient Machine Learning on Embedded Devices
27mAdvancements in optimizing ML Inference at CERN
32mDeploying & serving optimized ML pipelines using graalvm
25mMachine Learning on Microcontrollers; Tiny, Useful and Fun
04mMachine Learning in C++ with PyTorch
39mLeveraging the power of C++ for efficient machine learning on embedded devices
53mMachine Learning Platforms