Apache Airflow @ Umuzi.org

on YouTube ↗36m

" summary

Umuzi uses Apache Airflow to replace brittle cron and ad hoc scripts that handled thousands of applicant records, moving to dynamic DAG generation (Python-created tasks in loops), templated operators (Bash, Python, Branch, Slack/email) and the web UI for run history and tracebacks. Deployment runs via Docker Compose deployment with Postgres metadata and Celery/Mesos queues, DAG files on a Git volume (git pull to update). Porting costs exist but ongoing result is faster, clearer recruitment pipelines and fewer manual runs.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 2:57Muzie's mission and growth
3. 9:30Data pipeline challenges
4. 10:13What is Apache Airflow
5. 14:29Why Airflow is cool
6. 18:40Interacting with Airflow
7. 20:11Airflow web UI demo
8. 24:45Airflow architecture
9. 26:31Q&A session

" related talks

• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 35mBecome a Conductor with Apache AirflowMarwen Ben Dhahbia · Devoxx France 2021
• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 34mData Ingestion in Python using Apache AirflowSowmya Ganapathi Krishnan, Yew Wee · PyCon APAC 2018
• 01h 18mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 24
• 26mGo with the Flow: Automating Your Workflows with AirflowLeo Guinan · PyOhio 2018
• 38mAirflow DAGs Made Easy: Introducing the Astro SDKDaniel Imberman, Tamara Janina Fingerlin · Python Web Conf 2023
• 52mBetter ETL with Managed Airflow in ADFNiall Langley · SQLBits 2024