Upgrade Tooling w/Spark: It's not just for language versions
summary
A set of migration rules converts common Spark 2 patterns to Spark 3 equivalents using ScalaFix rules for Scala and Bowler interactive mode for Python, swapping RDD json reads to session.createDataset plus a String encoder and adding imports via a custom util. Running it on spark-testing-base produces a real PR but remains a proof of concept with printlns, an unfinished accumulator rule, frequent Python misapplies due to lack of types, and struggles against a 548-line exclude list of removals.
AI-written summary · about
chapters (15)
- 0:00Start
- 1:35How I Came To Spark
- 2:38Open Source Live Streams
- 4:32Upgrading Spark
- 13:14Data Frame Reader
- 15:00Interactive Mode
- 17:26Adding Spark 3 Support To Spark Testing
- 18:11Upgrading the Sql Context Constructor
- 18:43Adding the Spark Session Import
- 20:24Rename Functions
- 21:11Scala Fix Rules
- 21:29Hive Context
- 21:54The Sql Context Constructor Rule
- 22:25Migrating the Deprecated Readers
- 23:26Add Import if Not Present
related talks
47mKeeping the “fun” in Apache Spark Datasets and FP
21mAvoiding Spark Pitfalls at Scale
01h 05mClosing panel
46mSpark: One Tool for Everything?
31mSpark challenges
47mScala: The Unpredicted Lingua Franca for Data Science
23mSpark Datasets: why they aren't great and what could be done
36mNo more struggles with Apache Spark (PySpark) workloads in production