Hyperrest: A new Apache Arrow API For High Performance Data...
summary
Hyperrest is an Arrow-based API implemented in Dremio that replaces row‑centric ODBC transfer with Arrow Flight style columnar transfer so Pandas avoids copies. Benchmarks show ODBC 31s, turboODBC 17s, and Flight 11s for the same JSON scan and fetch. The demo uses Pandas, pyarrow and Dremio, demonstrates that schema negotiation lets clients read buffers directly with no serialization, and the new API materially reduces transfer and conversion overhead.
AI-written summary · about
chapters (7)
- 0:00Hyperrest and data pipelines
- 4:58Control messages in data mesh
- 9:14Data transfer and serialization
- 16:48Apache Arrow as unified format
- 24:42Columnar data layout benefits
- 31:04Arrow Flight adoption in systems
- 36:53Q&A
related talks
33mArrow in Flight - New Developments in Data Connectivity
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
35mPractical Applications of Apache Arrow
01h 04mWhere We’re Going, We Don’t Need Rows: Columnar Data Connectivity with Apache Arrow ADBC
36mEmbrace the chaos: Composable data systems with fewer asterisks
31mODBC takes an Arrow to the knee: ADBC
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
41mFaster SQL with pandas and Apache Arrow