Query like a Pro: ES|QL meets Apache Arrow and Pandas
summary
ES|QL runs inside Elasticsearch and streams results as Apache Arrow so Pandas can ingest them without JSON serialization, demonstrated by a one‑liner that maps ESQL Arrow output to a DataFrame. Demo loads NYC taxis from Parquet into Elastic Cloud serverless, samples down to <100k sampled rows, runs piped ES|QL queries (source → eval → bucket → sort) and plots results in matplotlib. Shows a simple anomaly rule P95 > 2.5× average and verifies plots and bucketing work end to end.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 2:02ES|QL as query engine
- 6:27Apache Arrow integration
- 9:54Demo setup and data loading
- 13:36Querying and analysis in pandas
- 15:45Visualization and insights
related talks
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
22mAccelerating Analytics with Apache Arrow
24mLeveraging the Apache Flight Python Client and InfluxDB
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
30mReconciling Kafka, SQL and Parquet