The State of Apache Airflow
summary
Claims heavy on community and numbers: 165 million downloads and ~2,000 contributors, Airflow 2.10 is current, demos show operator-based orchestration, Amazon provider and MWAA support, and a long list of practical optimizations — move imports into task scope, split huge DAGs, use a dag factory pattern, stagger schedules, and reserve pools for backfills. Newer features include OpenTelemetry metrics, defer operators to free worker slots, TaskFlow API improvements, hybrid executors and upcoming Airflow 3 with dag versioning and a slim API server.
AI-written summary · about
chapters (22)
- 0:00Apache Airflow intro
- 3:40Airflow popularity & community
- 6:26Airflow processing mistakes
- 9:16Orchestration vs task processing
- 11:04Optimizing workflows with Python imports
- 12:51Avoiding complex Python workflows
- 13:56DAG factories for efficient workflows
- 15:01Scheduling with random offsets
- 16:28Tuning Airflow performance
- 17:30Managing backfill jobs
- 19:19Monitoring & metrics in Airflow
- 21:06TaskFlow API benefits
- 22:10Improving task execution with defer
- 23:56Cleaning metadata database
- 24:17Automating Airflow deployments
- 25:22Enhancing developer experience with local setups
- 26:47AI for DAG development
- 27:53Future: hybrid executors
- 28:36Recent Airflow features
- 32:30Airflow 3 improvements
- 34:17Staying updated with community
- 35:22Q&A
related talks
32mWhat's coming in Apache Airflow 2.0
22mOrchestrate your data pipelines using Apache Airflow
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
25mRunning Apache Airflow at Scale
36mElegant data pipelining with Apache Airflow
43mHelp! My data is late!
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
36mApache Airflow in Production - Bad vs Best Practices