From Tensors to Clouds — A Practical Guide to Zarr V3 and Zarr-Python 3
summary
Zarr V3 and Zarr-Python 3 formalize interoperability and extensibility: a new abstract Store ABC lets users register custom stores (examples include git-like and Rust-backed stores), codecs move to an entry-point system so only Blosc/Zstd/Gzip ship by default, sharding groups many chunks into shards to fix S3/GCS latency, GPU support currently reads arrays into GPU memory but decodes on CPU, async IO is supported, demo converts a PNG logo to Zarr successfully, and 3.0.7 ships with an info/infocomplete split and an auto chunking heuristic that is convenient but not always optimal.
AI-written summary · about
chapters (8)
- 0:00Zarr V3 introduction
- 1:13Zarr fundamentals
- 5:07Motivation for Zarr V3
- 7:13Zarr-Python 3 design updates
- 13:35Zarr V3 features
- 19:00Performance optimization
- 20:47Demo: image to Zarr
- 21:56Q&A
related talks
21mCloud-Native Data in Practice
23mLessons learned building a cloud native geospatial data management system
29mZstandard in Python 3 14 Faster Compression You Can Use Today
28m10 years of Parquet- what's next?
32mBeyond Zstandard: The Future of Data Compression
09mLightning Talk: Model Checkpoint Compression With OpenZL
22mKubernetes Meets Climate Data: Designing Analysis-Ready Dat...
15mAndrew Zane, Unstructured | Databricks Data + AI Summit 2024