STACD: STAC Extension with DAGs for Geospatial Data and Algorithm Management

on YouTube ↗14m

" summary

Proposes STACD, a STAC extension that attaches DAGs, algorithm versioning and execution metadata so workflows are first-class catalog entries; provides a reference implementation using Apache Airflow that reads YAML DAGs, spins algorithms in Docker, and writes assets to Google Earth Engine. Demonstrates selective recomputation (re-run only affected nodes), on-demand lineage construction and visualization, parallel execution of raster/vector pipelines, and successful end-to-end runs that create dated dataset assets—full area processing takes about a day.

AI-written summary · about

" chapters (6)

1. 0:00STACD introduction
2. 1:22Community planning tools demo
3. 6:59STACD workflow config
4. 8:24STACD DAG execution demo
5. 11:34Future plans for STACD
6. 12:37Q&A

" related talks

• 29mDagster: a Framework for Data Processing ApplicationsNick Shrock · Scale By The Bay 2019
• 15mSPADE: Signal-Aware DAG Scheduling and Dynamic Provisioning for Data Processing ClustersAdam Lechowicz, Rohan Shenoy, Noman Bashir, Adam Wierman, Christina Delimitrou, Mohammad H. Hajiesmaili · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)
• 47mDagster: An open source Python library for building data applicationsNicholas Schrock · Crunch Conference 2019
• 38mAirflow DAGs Made Easy: Introducing the Astro SDKDaniel Imberman, Tamara Janina Fingerlin · Python Web Conf 2023
• 24mManaging Heterogeneous Spatial Data with Analysis Ready Cloud Optimized formatsLeo Ghignone · FOSS4G SotM Oceania 2023
• 46mUse frameworks to test Apache Airflow data pipelinesTamara Janina Fingerlin · Conf42 Python 2024
• 26mCI/CD For Data – Building Data Development Environments With Open Source StacksAdi Polak · Big Data LDN 2022
• 24mAsset based orchestration: the future of data orchestrationJonathan Neo · Data Teams Summit 2023