Scoring ONNX ML Models with Scala
summary
Agate is a pure-Scala library that scores ONNX models on the JVM, used at Stripe for batch scoring with Spark and Scalding and for low-latency real-time services; it implements ~160 ONNX operators incrementally (example model had 6 distinct operators with 2 unsupported, ~25% missing), avoids Python in production, scores many models in a few milliseconds without GPU, uses memory-mapped embeddings and Graal native-image binaries for fast CLI and Python interop, and verifies correctness by replaying training data.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 3:58What is ONNX and its structure
- 8:12Introducing Agate library
- 9:15Designing the tensor class
- 13:11Shape and storage abstractions
- 17:43Implementing ONNX operators
- 19:51Dynamic vs static scoring
- 22:39Demo of Agate CLI tool
- 27:06Future of ONNX and Agate
- 28:11Q&A session
related talks
05mONNX Runtime
39mMachine Learning with Scala on Spark
40mScala and Spark in the New Deep Learning Era
28mBring the Power of ONNX to Spark as it Never Happened Before
45mLeverage Power of Machine Learning with ONNX
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
32mMachine Learning Infrastructure at Stripe: Bridging from Python JVM
30mScalable ML acceleration with ONNX Runtime