Lessons Learned: Ultra-Fast Feedback with spark
summary
Presents an approach to speed up Spark edit-test cycles by collapsing repeated MapReduce-style joins into cogroup-based stage collapse and reusing stable Parquet checkpoints, reporting examples like a web-analytics query falling from ~6.0s to ~0.4s and pipeline replays shrinking from months to hours or days. Describes using the Spark driver architecture with RDD/Dataset transforms and local joins, mentions Catalyst limitations with nested structures, and highlights the Centrifuge tool for fast incremental regression checks.
AI-written summary · about
chapters (12)
- 0:00Start
- 0:52Data Engineering?
- 1:04Data operations can be simplified!
- 1:49Data devs are not easy to simply
- 2:14Big Data Devs SLOW
- 4:05CoGroup API
- 5:14An example: Web Analytics
- 5:25Checkpointing the pipeline
- 5:50The Code (for reference)
- 6:02How many stages could we skip?
- 6:59What do we get?
- 7:48Going Further
related talks
55mIntroduction to Apache Spark using Frameless
43mImproving PySpark Performance: Spark performance beyond the JVM
40mBest Practices and Optimization with the Spark Engine
30mThe Future of Scala Performance on Big Data
19mHow to accelerate Apache Spark pipelines on Amazon EMR with RAPIDS
39mAccelerating Data Science with RAPIDS
38mCharacterizing the performance of Big Data processing platforms
30mUsing Row Groups for fast filtering of large parquet files