Dealing with the versioning of production-ready models
summary
Company runs Toon® on Databricks and Spark, serving predictions at scale and tracking ~500 models with eight active deployments while processing ~1 TB/day and storing ~1 PB. Pickled Python models break across runtime upgrades, so options considered include ONNX portability, containerizing via MLFlow Projects, or periodic retraining. The pragmatic choice is scheduled retraining combined with pipeline integration and MLFlow registry tracking, reducing version lock without rewriting custom signal transformations.
AI-written summary · about
chapters (13)
- 0:00Welcome!
- 1:12Problem Statement
- 2:38Company Introduction
- 5:23Design Architecture
- 7:16Machine Learning Flow
- 8:56Pickling
- 13:29Solving Pickling Issue
- 14:26Possible Solutions
- 15:501. Portable Format (ONNX)
- 17:562. Containerize
- 19:503. Periodically Retrain
- 22:28Key Takeaways
- 23:35Questions
related talks
42mDeep learning beyond the learning
29mScaling to Millions of ML Models to Solve the Problems of SRE and Security
17mProductionizing Data Science Workloads
51mSpeeding up deployment of machine learning models
46mProduction Model Deployment
29mMachine learning in production
33mDeploying and Scaling Spark ML and Tensorflow AI Models
35mGetting Machine Learning Models Ready For Production Using Python