The Promise of Hierarchical Indices for Data Beyond 2 Dimensions
summary
Claims that the pandas Panel deprecated leads to choices and compares actual tools: Panel, xarray, pandas MultiIndex and static_frame HLock. Presents concrete benchmarks (500 iterations on pandas 0.23.4), finds Panel often fastest, xarray similar, static_frame competitive for 2‑D but much slower for 3‑D hierarchical indices. Builds code to parse NOAA buoy data, uses mean plus std thresholds to detect events and confirms a Dec 17–18 massive swell across multiple buoys.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:25Need beyond two dimensions
- 10:50Hierarchical indices
- 16:04Real-world data with hierarchy
- 21:203D performance comparisons
- 27:04Deprecation and future options
- 29:32Case study: massive swells
- 30:15Q&A
related talks
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
34mNoSQL Python: making data frames work for you in a non-rectangular world
01h 20mAdvanced Data Storage
28mWhy you should stop pretending your sparse data is dense
32mDataFrames: scaling up and out
39mMastering DataFrame Diffing Techniques
27mInequality Joins in Pandas with Pyjanitor
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars