A 101 in time series analytics with Apache Arrow, Pandas and Parquet
summary
Shows how a production pipeline is built end to end: a machine simulator → MQTT → Telegraf ingestion → InfluxDB 3.0 storing data as Parquet compression and exposing results via Arrow Flight SQL and DataFusion. Demos use PyArrow and Pandas to downsample and aggregate, write/read Parquet, and feed Plotly/Grafana dashboards. Notebook examples include anomaly-detection toolkit runs and a NOAA water-temperature autocorrelation test that returns no significant autocorrelation. Claims large storage and scan savings after switching to Parquet and Arrow-backed tooling.
AI-written summary · about
chapters (16)
- 0:00Introduction to time series analytics
- 1:26Why time series databases matter
- 5:34Time series database architectures
- 7:19Telegraf ingestion agent overview
- 9:02SQL aggregate functions in InfluxDB
- 9:43Apache Parquet explained
- 11:48Apache Arrow framework basics
- 13:12Arrow Flight and Flight SQL
- 15:38Arrow table creation example
- 17:24Working with Parquet files
- 18:28Downsampling data with Pandas
- 20:13Sending data to InfluxDB
- 21:16Anomaly detection toolkit
- 22:59Auto correlation explained
- 26:06Resources and tools summary
- 28:44Q&A session
related talks
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
22mAccelerating Analytics with Apache Arrow
46mEfficient Data formats for Analytics with Parquet and Arrow
24mLeveraging the Apache Flight Python Client and InfluxDB
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
01h 02mApache Arrow: High-Performance Columnar Data Framework