The ecosystem of geospatial machine learning tools in the Pangeo world
summary
Cloud-native stack proves concrete: kvikIO/quickiio reads Zarr straight to GPU via NVIDIA GPU Direct Storage (nvcomp needed for compressed chunks), xarray does lazy multidimensional slicing, and zentri composable data pipelines convert stacks to PyTorch tensors while handling multi‑CRS and mixed resolutions. Benchmarks show ~25% faster reads; an 18 GB→1 TB training example run for 100 epochs drops from ~24 to ~18 hours/day (≈6 hours saved/day, ~180 hours/month), roughly NZD900/month in estimated savings.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:39Main 3 things
- 2:52What is Pangeo
- 4:25Pangeo ML Stack
- 4:58Pangeo Ecosystem
- 7:59Examples
- 9:05Nvidia GPU Direct Storage
- 11:16Xure
- 13:17C
- 15:52Zri
- 16:48Results
- 18:04Resources
- 19:05Questions
related talks
31mTraining AI Models on Massive Datasets
19mScalable Data and AI Solutions with Cloud-Native Architecture
20mCloud Native Patterns for Geospatial Analytics and AI
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
36mAccelerating High-Performance Machine Learning at Scale i...
18mUnleash your Private Datasets to Power up AI
41mAccelerating Data Science with RAPIDS
28mA pythonista’s pipeline for large scale geospatial analytics