Bridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
summary
A new service replaces JDBC+Spark with Arrow Flight and DuckDB to stream Parquet/Hudi data as Arrow buffers directly into Pandas/Polars, avoiding column-to-row pivots and serialization. The point-in-time joins run inside DuckDB so training snapshots assemble server-side; benchmarks show a 100M-row, 7-column join dropping from ~129s (Spark) to ~8s, and typical 1 GB extracts that once took ~20 minutes now approach network-line speed. The service supports concurrent clients for the Hopsworks feature store and falls back to Spark for larger-than-memory jobs.
AI-written summary · about
chapters (11)
- 0:00Introduction and motivation
- 1:15Lakehouse and feature stores
- 4:44Why Arrow Flight and DuckDB
- 5:49Arrow in-memory format explained
- 8:17Arrow Flight for data transfer
- 11:08DuckDB for efficient joins
- 12:13DuckDB in Python
- 14:39Flying Duck architecture
- 17:25Enterprise challenges and solutions
- 21:39Benchmark results and performance gains
- 24:28Future improvements and conclusion
related talks
33mArrow in Flight - New Developments in Data Connectivity
28mAnalyze data easily with duckdb - and the implications on data architectures
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
34mData Wrangling with DuckDB
16mAirport for DuckDB: Letting DuckDB take Apache Arrow Flights
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
01h 02mApache Arrow: High-Performance Columnar Data Framework