Designing for high-performance execution with Arrow and Polars
summary
Claims concrete speedups: Arrow’s zero-copy Arrow format and Polars’ lazy execution cut serialization and data-movement costs and often give order-of-magnitude runtime improvements on analytical pipelines. Examples include end-to-end columnar historical VaR and production liquidity aggregation where memory layout, cache locality, and SIMD-compatible layouts turn noisy loops into vectorized scans. Real-world survey finds only 2 of 9 textbook VaR examples use columnar code. When loops are unavoidable, JIT tools like Numba vectorize them to acceptable throughput.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:47Morgan Stanley work
- 6:00Columnar storage benefits
- 10:13Arrow history
- 13:46Polars vs loops
- 16:32Hardware for performance
- 22:18Cache locality and SIMD
- 26:36Vectorization and CPU
- 29:45Polars execution models
- 30:49Portfolio risk with columnar
- 32:55When loops are needed
- 35:22Zero-copy with Arrow
- 37:28Key takeaways and final recommendations
related talks
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
34mLeveraging Apache Arrow for ML Workflows
39mTaming the language border (...)
46mInternals of Speeding up PySpark with Arrow
31mThe Arrow revolution in pandas and Dask
32mThe PyArrow revolution in Pandas
27mApache Arrow - The columnar format! What's that?
37mPolars 1.0 and beyond