An Apache Spark query's journey through the stack
summary
Query plan construction and constant folding by Catalyst, then file-level pruning via Delta Lake/Parquet metadata, determine how much data Spark actually reads; task count maps to partitions (~128MB chunks) and one task per executor so sharding matters; expensive operations are sorts and shuffles (spill to disk costs money), while broadcast joins avoid shuffles; newer C++ engines (Gluten/Velox / Photon style) plug into Spark and claim 2–3× speedup on the same hardware versus JVM execution.
AI-written summary · about
chapters (9)
- 0:00Spark and Lakehouse intro
- 4:05Spark architecture basics
- 9:30Query planning with Catalyst
- 13:24Predicate pushdown & pruning
- 15:54Cost estimation & joins
- 17:58Query execution & scheduling
- 20:09Shuffling & broadcast joins
- 23:03Query journey summary
- 25:49Q&A on Gluten & Velox
related talks
15mOptimizing Query Performance in Big Data Systems
41mOptimising Apache Spark and SQL for Improved Performance
50mChallenges that everyone struggles while productionizing Apache Spark workloads
40mBest Practices and Optimization with the Spark Engine
43mBeyond Shuffling: Scaling Apache Spark
40mScaling with Apache Spark (or a lesson in unintended consequences)
04mKuassi Mensah Interview
28mAI-driven query plan optimization in big data processing tools