Extending Spark Machine Learning Beyond Linear Regression
summary
Shows how to add custom Transformer and Estimator stages to Spark ML pipelines in Scala (Python bindings not supported yet), with runnable examples: a simple word‑count Transformer and a Naive Bayes classifier that trains on feature vectors. Explains that iterative training currently switch to RDD and cache for performance, exposes Spark’s parameter system so hyperparameter grid search and cross‑validation work, and covers model export (MLReader/MLWriter, custom save) plus publishing code to GitHub, Maven Central and spark‑packages.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:15Spark MLA
- 3:54Introduction
- 5:19The Good Stuff
- 6:43Spark Datasets
- 12:05Spark ML Adventure
- 12:32Building a Transformer
- 12:53Transform Schema
- 14:16Word Count
- 14:56Verify Input Schema
- 15:26Verify Transform Schema
- 15:43Pipeline Stage
- 16:18Configurable Parameters
- 17:03Crossvalidation
- 17:14Example
- 18:57Input columns
- 20:00Classifier
- 20:36Naive classifier
- 21:08Vector assembler
- 22:20Dataframe transformations
- 22:54More code
- 23:07Basetrade cadets
- 24:32Lunch
- 24:46Putting it in an ML pipeline
- 26:27Publishing your ML model
- 27:27Spark example repo
- 28:19Spark books
- 29:08Buy this book
- 30:10Data and Comps
- 30:57Questions
related talks
39mMachine Learning with Scala on Spark
23mRethinking scalable machine learning with Spark ecosystem
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
44mApache Spark and ML workflows
35mPySpark : Combining Machine Learning & Big Data
33mUsing Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software
50mApache Spark vs Machine Learning
31mFrom Pandas to Spark