Machine Learning with Scala on Spark
summary
Claims that Spark.ml makes production ML easier by letting teams run hyperparameter tuning with parallel Spark runs (spin 100 boxes instead of waiting weeks), do data cleaning and ETL in the same environment, and deploy typed pipelines where models and transformers share an API. Examples: ported Kaggle grant prediction and a VQA image Q&A pipeline (state of the art ~70%), built a smile-recognition demo that launched a company. Gaps: limited deep‑learning choices, manual categorical indexing, tree depth capped at 30, and immature graph frames.
AI-written summary · about
chapters (46)
- 0:00Introduction
- 1:12Definition of Machine Learning
- 1:36Why talk about Machine Learning
- 2:59Not happening is bad
- 3:17Prediction
- 3:49Software Industries
- 4:43Machine Learning
- 9:36Why do people do this
- 11:13Why is DSR doing this
- 11:58Python
- 12:12Scala
- 12:24Scala vs Spark
- 12:56Machine Learners
- 13:52Python vs Scala
- 14:51Kratt
- 15:43Menial Work
- 18:36Lack of Understanding
- 19:01Spark
- 19:35Tabriz
- 20:08DataFrame vs DataSet
- 21:12Scala as the defacto ML language
- 21:57Problem with onboarding
- 22:51How we assimilate the influx
- 23:55Twitter
- 24:21Hire a person
- 24:41MOOCs
- 25:30No spec by spec
- 26:54When will it work
- 27:57How does it feel
- 29:31The point of Spark
- 30:27Why Spark
- 32:07Hyperparameter Tuning
- 32:53Data Cleaning
- 33:06ETL
- 33:37Migration
- 34:30Catching Errors
- 35:01Scale
- 35:39Picture
- 36:25categorical variables
- 36:48pipelines
- 37:05pipeline approach
- 37:20graphics staff
- 37:47pros
- 38:21disappointments
- 38:54conclusion
- 39:06thank you
related talks
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
50mApache Spark vs Machine Learning
17mMachine Learning at the Speed of Light with Spark
44mApache Spark and ML workflows
38mFantastic ML apps and how to build them
23mRethinking scalable machine learning with Spark ecosystem
04mThe machine-learning renaissance
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers