Where We’re Going, We Don’t Need Rows: Columnar Data Connectivity with Apache Arrow ADBC
summary
ADBC is a multi‑language client API and driver standard that delivers query results in Apache Arrow columnar format instead of row rows, eliminating the expensive row↔column shuffle and cutting CPU and memory overhead. It ships as about a dozen drivers and bindings for ~10 languages and plugs into pandas, polars, DBT Fusion and PowerBI; true end‑to‑end paths use Flight SQL while wrapped drivers give a slower fallback. Examples include a DuckDB shared‑library driver, columnar transfer and zero‑copy delivery that demonstrably speeds result movement.
AI-written summary · about
chapters (15)
- 0:00Columnar data connectivity intro
- 1:03Arrow and ADBC overview
- 8:52Legacy row standards
- 13:51Columnar vs batch storage
- 16:20Row transfer inefficiencies
- 18:05Why columnar formats
- 22:23ADBC architecture
- 25:56ADBC limitations
- 32:36ADBC drivers and ecosystem
- 35:23ADBC terms and manager
- 41:27ADBC with DuckDB
- 45:39ADBC future 2030
- 49:16How to contribute
- 51:44Q&A: type impedance
- 57:02Q&A: compression & performance
related talks
33mArrow in Flight - New Developments in Data Connectivity
31mODBC takes an Arrow to the knee: ADBC
35mPractical Applications of Apache Arrow
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
27mApache Arrow - The columnar format! What's that?
32mThe expanding Apache Arrow universe
41mFaster SQL with pandas and Apache Arrow
36mEmbrace the chaos: Composable data systems with fewer asterisks