AutoML on Spark: The Myth That Became Reality
summary
A team presents LightAutoML, their AutoML system (1.3K GitHub stars, 150K+ PyPI downloads), and explains the challenges they faced when porting it to run on Spark with Scala/Java. The talk covers why AutoML matters today and concrete technical issues encountered while adapting an existing Python-based AutoML solution to a Spark+Scala environment.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 5:20What is AutoML
- 8:13Spark as foundation
- 13:13RDD and DataFrame
- 20:30MapReduce pattern
- 24:29Gradient descent in ML
- 27:01AutoML pipeline
- 30:59AutoML transfer issues
- 36:42Feature engineering
- 41:02Categorical data handling
- 42:063D parallelism
- 45:45Summary
related talks
39mMachine Learning with Scala on Spark
08mBroadening AI Adoption with AutoML
38mFantastic ML apps and how to build them
17mMachine Learning at the Speed of Light with Spark
50mApache Spark vs Machine Learning
23mRethinking scalable machine learning with Spark ecosystem
28mAutoML—hype vs. value
48mUsing Automated ML models in Azure Synapse Analytics