The future of data: Apache Spark 3.1, Delta Lake and Koalas
summary
Apache Spark 3.1 introduces adaptive execution that can change join strategies to broadcast, reduce shuffle partitions, and split skewed partitions at runtime, plus SQL dynamic filtering and GPU support. Delta Lake provides ACID transactions, time travel and Parquet-based transaction logs so streaming and batch workloads can run safely together. Koalas is being integrated into Spark as Pandas APIs on Spark (targeted for Spark 3.2), and Databricks' Photon C++ vectorized runtime promises about 3× CPU speedups.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 6:45Spark 3.1 updates
- 12:00Koalas project
- 17:36Classic architecture issues
- 20:00Delta Lake storage
- 27:06Lake House evolution
- 33:08Lake House architecture
- 42:54Q&A
related talks
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
42mAn Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric
31mUsing Spark to accelerate your lakehouse architecture with Fabric
58mBuild reliable data lakes with Delta Lake & Databricks
39mDelta Lake: Diving into Data Lakes Without the Downsides
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
30mKoalas
38mGetting Started with Delta Lake