Building Large Scale ETL Pipelines with Dask
summary
New Dask DataFrame implementation claims concrete wins: switching to pyarrow-backed strings cuts string-heavy memory use by over 50% and speeds some string ops 2–3x, a redesigned shuffle achieves constant-memory shuffling and handles tens of terabytes using 500 GB RAM, and a query optimizer yields up to a 19x speedup (950s to 50s) on benchmarks. The team assembles an ETL pipeline reading JSON from S3, writing to Delta Lake, runs on a 250-worker cloud cluster, and the demo shows the new path completing while the old one spills and stalls.
AI-written summary · about
chapters (13)
- 0:00Dask for ETL pipelines
- 1:10Dask DataFrame improvements
- 3:17Arrow strings for memory
- 4:22Distributed join performance
- 5:47Shuffling for scaling
- 7:36Query optimization in Dask
- 10:05Small partitions handling
- 11:31Reducing network transfers
- 13:59Demo old vs new
- 17:05Dask in large ETL
- 21:47Dask with workflow schedulers
- 22:30ETL with JSON data
- 24:38Q&A Dask vs Spark
related talks
31mThe Arrow revolution in pandas and Dask
35mData Processing Pipelines for Small Big Data
26mDask-SQL: Empowering Pythonistas for Scalable End-to-End Data Engineering
32mDataFrames: scaling up and out
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
18mETL at Scale: Optimizing PySpark + Airflow Workflows
30mSpeed is Not All You Need for Data Processing
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows