Using Zarr for drug discovery datasets in Polaris
summary
Polaris uses Zarr as the canonical store for ML-ready drug discovery data, claiming one format handles kilobytes to terabytes and modalities from SMILES to 3D conformers. The team builds a Hub with converters, a dataset factory, and PyTorch/TensorFlow data loaders; they report a surprising 3.6 billion‑row submission that breaks the in‑memory assumption. Optimisations include Zarr arrays chunked and compressed, single data point per chunk for local disk random access, and signed‑URL/auth strategies for private datasets.
AI-written summary · about
chapters (6)
- 0:00Polaris and drug discovery
- 8:14Polaris for benchmarks
- 13:58Zarr for data storage
- 23:52Zarr in Polaris datasets
- 28:50Zarr challenges
- 32:01Conclusion and involvement
related talks
28mThe Beauty of Zarr
31mAccelerating Single-cell Bioinformatics with N-dimensional Arrays
31mExplore, annotate, and analyse multidimensional image data with napari
24mRivian: Scaling Data & ML Platforms
23mThe ecosystem of geospatial machine learning tools in the Pangeo world
21mPolars: A highly optimized dataframe library
13mZML: ML framework for Zig
37mPolars 1.0 and beyond