Zarr vs. HDF5
summary
Direct comparison of actual behavior and extensibility: Zarr says 2015, pure Python, codecs via numcodecs (Blosc, JSON, msgpack), and multi‑threaded reads with S3/LMDB/zip backends, while HDF5 is ~20 years old, C core, single‑threaded default, VFD virtual file driver with a contiguous 64‑bit address space and C filters registered by integer IDs. Examples: a delta‑delta codec and an S3 file driver exist; HDF5 extensions require compiling or malloc‑compatible native code and face ABI/threading limits.
AI-written summary · about
chapters (8)
- 0:00Zarr vs HDF5 intro
- 8:57Age and language support
- 16:40HDF5 custom filters
- 29:58Zarr custom filters
- 39:51HDF5 storage backends
- 42:16Zarr storage backends
- 45:29Key takeaways
- 50:46Q&A
related talks
36mStore and manage data effortlessly with HDF5
31mAccelerating Single-cell Bioinformatics with N-dimensional Arrays
01h 20mAdvanced Data Storage
32mBeyond Zstandard: The Future of Data Compression
30mWhat a serverless database means for users
33mThe Evolution and Innovation of CubeFS
22mQZFS: QAT Accelerated Compression in File System for Application Agnostic and Cost Efficient Data Storage
25mMulti Dimensional Clustering with Z Ordering