When DataFrames fail, resort to mapPartitions #bbuzz
summary
DataFrame APIs hit nested-schema and lineage limits: a generated schema spans ~2,000 lines with nine-deep, 11-wide nesting and exploding 10 GB of source into ~120 GB and ~40M output entities. Rewriting the logic into mapPartitions that runs batched SQL (typically batching 1,000 IDs with IN queries), reduces remote query count and controls writes to Cassandra, trimming runtime from 96h to 3.5h (about 20x faster) while fully utilizing cluster CPU.
AI-written summary · about
chapters (7)
- 0:00Introduction and context
- 4:38Spark data partitioning issues
- 9:59Data locality and task distribution
- 14:12Checkpointing and optimization techniques
- 19:15When DataFrames fail
- 23:11Performance results and trade-offs
- 26:13Q&A session
related talks
34mAvoiding Big Data Antipatterns
34mDimensional Data Modeling in the Modern Era
40mProductionizing Big Data - Stories from the Trenches
29mDataEngBytes 2023 - BRI-T1-03 - Yousry Mohamed
21mAvoiding Spark Pitfalls at Scale
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
40mSpark Analytics on Cassandra Data
26mFrom datasets to tables in a multitenant data lake