You Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
summary
Claims that many lakehouse tasks don’t need Spark for routine work and that modern Python stacks handle them: use delta lake Python library (create/write/merge/inspect history and time travel), Polars/Arrow for dataframes, and DuckDB for fast local queries (read-only on Delta today), with the surprising practical limit of almost 100 gigabytes on a commodity laptop. A small open source tool, lectower, wires catalog + DuckDB/Delta for CLI and web exploration and demonstrably runs the demos shown.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:10What is a data lakehouse
- 6:50Why Spark may not be best
- 8:35Python alternatives to Spark
- 10:02Demo with Delta Lake Python
- 17:30Reading Delta Lake tables
- 18:57Delta Lake time travel
- 20:20Delta Lake with DuckDB
- 22:29Introducing Lectower
- 26:28Conclusion
related talks
31mHow to quickly build Data Pipelines for Data Scientists
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake
25mBuild open format data lakehouse
28mHigh speed data from the Lakehouse to DataFrames with Apache Arrow
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
30mSpeed is Not All You Need for Data Processing
35mData Processing Pipelines for Small Big Data
18mDelta Lake – The Foundation Of Your Lakehouse