Blosc2: Fast And Flexible Handling Of N-Dimensional and Sparse Datasets
summary
Blosc2 is a C and Python library plus a simple container format that claims up to 5–8× faster slicing via a double data partition and extreme compression (an 8 trillion‑cell Gaia cube shrinks from ~8 TB to under 2 GB, >4,000×). Dynamic plugins load shared libraries at runtime so new codecs work without rebuild; an example JPEG 2000 via Gro plugin yields good lossy image reduction. Automated tuning with B tune searches codec/filter/thread/partition tradeoffs and reports best combos by speed and ratio.
AI-written summary · about
chapters (10)
- 0:00Introduction to Blosc2 and its features
- 4:24HDF5 integration gains
- 6:52Sparse arrays compression
- 9:22C library multi-language
- 10:05Dynamic plugins
- 12:38Blosc2Gro JPEG 2000
- 14:23B-tune compression tuning
- 16:13B-tune genetic & DL modes
- 18:00B-tune real-time predictions
- 20:29Conclusion
related talks
25mNew Trends In Storing Large Data Silos With Python
01h 27mMastering Large NDArray Handling with Blosc2 and Caterva2
43mNew Computer Trends and How This Affects Us
36mScale your data, not your process: Welcome to the Blaze ecosystem
28m10 years of Parquet- what's next?
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
57mGraphBLAS: Building a C++ Matrix API for Graph Algorithms
32mBeyond Zstandard: The Future of Data Compression