Accelerating Single-cell Bioinformatics with N-dimensional Arrays
summary
Claim: parallelize single-cell workflows across Spark Dask Pywren and Apache Beam and store arrays as Zarr instead of HDF5. Implements Zarr format in scanpy, adds Spark bindings and a netcdf Java path to read HDF5 from cloud stores. Ships a numpy-like library zappy with pluggable backends, runs benchmarks showing near-linear speedup with overhead, and handles datasets up to 1.3 million cells by chunking and remote reads.
AI-written summary · about
chapters (8)
- 0:00Single-cell bioinformatics intro
- 2:30Why single-cell sequencing matters
- 8:40Wet lab to dry lab processing
- 13:23HDF5 storage challenges
- 19:03Zarr for cloud-friendly arrays
- 22:14Zarr and parallel backends
- 24:46Future of distributed analysis
- 26:14Q&A
related talks
33mUsing Zarr for drug discovery datasets in Polaris
39mAccelerating Data Science with RAPIDS
01h 20mAdvanced Data Storage
36mStore and manage data effortlessly with HDF5
34mLeveraging Apache Arrow for ML Workflows
25mScaling Up Genomics with Spark
01h 01mParallel Python Analyzing Large Data Sets
26mWho Said Wrangling Geospatial Data at Scale was Easy?