Managing Heterogeneous Spatial Data with Analysis Ready Cloud Optimized formats
summary
Built a fit‑for‑purpose Data Management System for the Reef 2050 RIMReP that centralises disparate sources into S3 using Zarr for multi‑dimensional rasters and GeoParquet for tables, converting NetCDF/GeoTIFF/CSV/Excel into ARCO formats so clients read only needed chunks via parallel libraries. Metadata lives in Frictionless data packages and drives generation of STAC and pygeoapi configs. Pipelines handle oddities (decoded value columns, long headers) and scale to several‑terabyte models and a table with 1,200 columns.
AI-written summary · about
chapters (8)
- 0:00Introduction to project
- 2:28Reef 2050 monitoring program
- 5:20Data types and formats
- 7:07Analysis ready cloud formats
- 11:23Metadata challenges and solutions
- 14:57Data pipeline and Excel example
- 20:18Team and contact information
- 21:00Q&A session
related talks
23mLessons learned building a cloud native geospatial data management system
22mTake your Business to a New Geography
20mCloud Native Patterns for Geospatial Analytics and AI
22mKubernetes Meets Climate Data: Designing Analysis-Ready Dat...
18mA Toolbox for Scalable Spatial Data Science in the Cloud
14mMosaic: A Framework for Geospatial Analytics at Scale
16mThe Open Data Cube and working towards a Digital Earth
34mGeoscience Big Data Computing with Python