Lessons learned building a cloud native geospatial data management system
summary
Builds a cloud‑native geospatial data management system for the Great Barrier Reef Marine Park Authority using Zarr, GeoParquet and Argo Workflows to ingest heterogeneous sources. Converts a 30 GB CSV and over 1 TB of netCDF into cloud formats; tuning Parquet row‑group size produces up to 5x query speed; limiting dictionary sizes shrinks one Parquet output from ~2 GB to 8 MB; multi‑dimensional chunking in Zarr reduces time‑series plot latency from minutes to seconds. Daily ingestion switches to conditional updates and an API fixes R interoperability.
AI-written summary · about
chapters (9)
- 0:00Introduction to the project
- 1:05Project overview and goals
- 3:39Code organization lessons
- 7:10Handling big data challenges
- 9:18Optimizing Parquet performance
- 14:44Zarr format and chunking
- 17:44Updating and maintaining data
- 19:10Ensuring interoperability
- 20:16Summary and Q&A
related talks
21mCloud-Native Data in Practice
24mManaging Heterogeneous Spatial Data with Analysis Ready Cloud Optimized formats
20mCloud Native Patterns for Geospatial Analytics and AI
22mTake your Business to a New Geography
36mAn Exploration in Storing Telemetry in Cloud Object Storage
22mKubernetes Meets Climate Data: Designing Analysis-Ready Dat...
14mMosaic: A Framework for Geospatial Analytics at Scale
20mOpen-source processing pipelines for Cyclone Gabrielle