Using Prefect to Build Geospatial Data Pipelines
summary
Australian Ocean Data Network describes replacing an ageing pipeline with Prefect to orchestrate geospatial ETL of NetCDF files, claiming easier maintenance, retries, scheduling and observability. Code lives in GitHub, runs locally in Docker during development and deploys to Prefect Cloud which triggers workers on AWS deployment to run validation, aggregation and product generation. Screenshots show flow logs, run history and a diagram of integrations; emphasis is on scalability and reduced ops overhead.
AI-written summary · about
related talks
28mBetter Python Coding with Prefect Blocks
24mManaging Heterogeneous Spatial Data with Analysis Ready Cloud Optimized formats
45mFrom telescope to data centre: adventures in astronomical data pipelines
16mEvolution of Extract-Transform-Load (ETL) processes towards data product pipelines
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
21mCloud-Native Data in Practice
19mpgFaaS: Geo-Processing made simple with Serverless technology
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...