Improve the efficiency of your Big Data..
summary
Claims that operating on compressed arrays yields real speed and memory wins: a 22 GB Central‑Europe precipitation dataset on an 8‑vCPU VM shows about 2× runtime improvement versus an uncompressed NumPy pipeline. The stack uses Caterva multi‑dimensional chunked containers, compression via Blosc2, and expression compilation with LLVM in IronArray, tunes chunk sizes to fit L3 cache, and produces compressed outputs with multi‑threaded decompression so the approach actually works at scale.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:28Multi-layered stack architecture
- 3:33Storage layer and Blosc compression
- 9:18Multi-dimensional data with Katana
- 15:45Computational layer and expression evaluation
- 22:08API layer and IronArray
- 23:13Katana Python demo
- 30:54IronArray computations demo
- 37:11Performance benchmarks
- 41:54Technology philosophy and future
- 42:38Q&A
related talks
01h 27mMastering Large NDArray Handling with Blosc2 and Caterva2
43mNew Computer Trends and How This Affects Us
25mNew Trends In Storing Large Data Silos With Python
26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...
01mSmall Big Data: using NumPy and Pandas
20mNumS: Scalable Array Programming for the Cloud
30m5 Steps to Speed Up Your Data-Analysis on a Single Core
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas