Connecting PyData to other Big Data Landscapes using Arrow and Parquet
summary
Shows how Apache Parquet and Apache Arrow actually interoperate: Parquet provides 16 kilobytes pages, row groups of ~100k–2M rows, column statistics and predicate pushdown so engines skip I/O; Arrow defines an in‑memory, zero copy columnar layout with C++ bindings and Python adapters. Examples include a Parquet C++ backend, Spark 2.3 moving DataFrames via Arrow and an observed transfer drop from seconds per million integers to about 0.05 seconds end to end, plus GPU and browser demos that work with Arrow buffers.
AI-written summary · about
chapters (10)
- 0:00Start
- 3:45A general problem
- 6:12Columnar Data
- 8:04Apache Parquet
- 9:47Why use Parquet?
- 12:25Predicate pushdown
- 14:32File Structure
- 16:46Read & Write Parquet
- 22:14Apache Arrow - Real life improvement
- 24:32GPU Open Analytics Initiative
related talks
24mPolyglot data with python: Introducing Pandas and Apache Arrow
46mEfficient Data formats for Analytics with Parquet and Arrow
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
22mAccelerating Analytics with Apache Arrow
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
33mFree Your Esoteric Data Using Apache Arrow and Python
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
33mArrow in Flight - New Developments in Data Connectivity