No more struggles with Apache Spark (PySpark) workloads in production
summary
Covers concrete fixes: use PyArrow to avoid Python/Numpy/Pandas serialization bottlenecks, prefer JDBC parallel reads with numeric lower/upper bounds and partitionCount to get a 7x speedup on a reported case, and use bulk-load APIs for fast writes. Shows how to tune executors, memory and partition counts, switch spark.scheduler for eager sections, and reduce shuffles with broadcast 10 MB thresholds, bucketing and partitioning. Demonstrates Airflow + spark_submit operator wiring for parameterized ETL.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 2:16Spark internals overview
- 8:23RDD vs DataFrame trade-offs
- 13:23Spark internals terminology
- 15:51Memory and resource tuning
- 19:49Parallel JDBC reads optimization
- 22:18Bulk load and write strategies
- 23:25Join performance techniques
- 25:56Case study and architecture
- 27:04Airflow integration for ETL
- 34:25Q&A session
related talks
30mImproving PySpark Performance: Spark performance beyond the JVM
21mAvoiding Spark Pitfalls at Scale
41mSpeeding Up Data Pipeline Construction in Apache Spark
52mSolving Analytical Problems using Apache Spark
43mOptimizing Spark applications from simple to complex with examples
24mTaming Data Skew in Production ML Pipelines
50mIntroduction to Real-Time Big Data with Apache Spark
30mOptimize Spark Jobs Using Its UI