Complex Machine Learning Pipelines Made Easy
summary
Salesforce describes a Scala library called Optimus Prime that sits on top of Spark and automates building thousands of multi-tenant, per-customer models by treating features as first-class objects. It runs type-aware transforms (tokenize, lemmatize, pivot), a single dot sanityCheck that removes label leakage and low‑variance features, and an AutoML-style classify that cross-validates across algorithms and parameter grids. The claim: pipelines train and predict per-customer at scale, producing production-quality models far faster than hand-coded Spark workflows on real customer data.
AI-written summary · about
chapters (15)
- 0:00Introduction to talk
- 1:21Salesforce data structure
- 4:09Machine learning with Optimus Prime
- 6:18Case classification use case
- 9:52Flexible pipeline design
- 13:46Feature vectorization process
- 14:30Statistical modeling overview
- 17:40Scalable pipeline implementation
- 19:06Optimus Prime advantages
- 20:14General pipeline approach
- 23:23Generic use case setup
- 25:33Automatic feature transformation
- 27:41Sanity check and feature selection
- 28:24Summary of Optimus Prime
- 29:36Q&A session
related talks
49mImpact on business and society
42mThe Lego Model for Machine Learning
39mMachine Learning with Scala on Spark
37mMachine Learning, The Right Way
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
37mAutomated Machine Learning: Building a Pipeline
23mRethinking scalable machine learning with Spark ecosystem
19mDrive transformation through machine learning