Doubt Truth to be a Liar: Non Trivialityy of Type Safety for Machine Learning
summary
Argues that getting compile-time type safety for feature vectors is hard and shows a concrete solution: Optimus Prime — a Scala library that layers typed features and Scala macros over Spark ML, using typed readers, feature builders, transformers and estimators, automatic name inference and DAG topological sorting to emit Spark pipelines. Demonstrates on Titanic data (75/25 split, cross‑validation) and productionizes into Salesforce Einstein, replacing ad‑hoc UDFs and mutable DataFrame overwrites and reducing late runtime failures.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:05A glimpse into the future
- 1:41Machine Learning 101
- 4:55Real-life ML
- 12:11Salesforce Einstein
- 13:54Predict survival on the Titanic
- 19:08Building ML application with Spark ML
- 33:49Types and interactions
- 36:45Feature Types V2
- 38:09Typed Features
- 40:03Feature names from vals using Macros magic
- 40:48Feature transformations with Implicit Classes
- 41:46Features and Transformers
- 43:31Typed pipeline stages
- 45:471-to-1 Transformer example
- 47:011-to-1 Estimator example
- 48:00Model training (reduce)
- 48:50Going forward with Optimus Prime
- 49:55Key takeaways
