Using DuckDBArrowFlight to Power a Feature Store
summary
A feature store prototype replaces Spark jobs with DuckDB + Arrow Flight to serve parquet files from a Hoodie lake using HDFS readers and a custom connector that discovers latest parquet files and pushes down partition filters. Training-data point-in-time joins run substantially faster — a demo read drops from 8s→1s — and training datasets can stream to clients with zero-copy Arrow Flight. Benchmarks show consistent speedups over Spark for small and moderate scale factors; the connector and HDFS read work in production.
AI-written summary · about
related talks
05mBuilding Tecton’s Feature Engineering Platform on DuckDB
40mDuckDB beyond the notebook
30mOLAP on Your Cassandra Data with Arrow, Flight SQL, ADBC, and DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
40mDuckDB: Harnessing in-process analytics for data science and beyond
34mLeveraging Apache Arrow for ML Workflows
34mData Wrangling with DuckDB
33mArrow in Flight - New Developments in Data Connectivity