`New` Workflow Orchestrator in town: "Apache Airflow 2.x"
summary
Apache Airflow 2.8+ and 2.9 are presented as a modern, evolving orchestrator with concrete results: 2,900 contributors, regular quarterly releases, and features like dynamic task mapping extended to whole task groups for massive parallel runs; deferred operators using async triggers that claim 80–90% fewer worker slots for many workloads; FS Spec integration for unified object-store IO; dataset-based scheduling plus open lineage with column-level provenance and forthcoming OpenTelemetry traces. New UI, unicode names, setup/teardown, notifiers and DBT-to-DAG mapping (Cosmos) are already shipping.
AI-written summary · about
chapters (12)
- 0:00Introduction to Airflow
- 2:29Core concepts and community
- 10:12Modern task development
- 16:22New functional features
- 18:20Data handling and scheduling
- 22:27Performance and automation
- 25:01AI, LLMs, and platform growth
- 27:20UI and observability revamp
- 29:25Platform extensibility
- 33:00Enterprise readiness
- 35:40Airflow 3 vision
- 38:23Q&A and closing
related talks
26mModern Data Orchestrators
40mAirflow 3 - the new beginning
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
35mBecome a Conductor with Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
43mHelp! My data is late!
06mComparing Argo Workflows and Apache Airflow at Scale