Going with the flow? Apache Airflow for operational-quality scientific workflows
summary
Uses Apache Airflow to turn existing Fortran/C and Python models into reproducible operational pipelines, integrating Conda environments, container images and CI/CD. Deployments go to AWS using MUA, ECR, ECS, Fargate, S3, EFS orchestrated by Terraform and GitHub Actions for lift‑and‑shift to partner agencies. Task grouping yields measurable speedup — reduced runtime from 3.8 to 1.5 minutes (42%) — and supports eReefs and Seasonal Streamflow Forecasting production workflows with clearer logging, retries and web UI.
AI-written summary · about
chapters (8)
- 0:00Apache Airflow introduction
- 4:37E-Refs project overview
- 7:26Why choose Airflow
- 9:11Airflow setup on AWS
- 10:35Final system architecture
- 12:40SETI workflow optimization
- 14:47Performance with parallel tasks
- 15:49Reflections and future work
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
16mBridging Data Science and Data Infrastructure with Apache Airflow
36mElegant data pipelining with Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
38mUse Apache-Airflow to Build Data Workflows
21mScaling and automating R workflows with Kubernetes and Airflow
01h 08mThe State of Apache Airflow