Updates on Spark, MLflow, and the broader ML ecosystem
summary
Overview of concrete updates: sparklyr 1.0 adds Apache Arrow support and reports a 40x read performance improvement for local installs; sparklyr 1.0.1 adds Delta Lake support so Spark gains ACID transactions and time-travel versions on data lakes. Demonstrated workflow writes a dataset to Delta, overwrites it, then reads the prior version back. Also covered Spark 3.0 preview (Java 11, GPU scheduling, binary file input), MLflow tracking for experiments, and community extensions that already exist.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:23Spark on R
- 3:23Whats been up to
- 4:29What is Delta Lake
- 5:23How to use Delta Lake
- 6:08Community Extensions
- 7:25Spark 30 Preview
- 8:32Barrier Execution
- 9:50MLflow
- 12:32Demo
- 15:00Book
- 15:24Mastering Spark
- 16:47Sparkler Joins the Linux Foundation
- 17:30Questions
related talks
59mSpark 2.0
54mWhat Does the Future Spark Have in Store for Us?
19mWhat's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0
10mNew Directions for Spark in 2015
25mCONTINOUS APPLICATIONS WITH SPARK 2.0
50mApache Spark vs Machine Learning
11mRecent trends in data and machine learning technologies
39mMachine Learning with Scala on Spark