Airflow 3 - the new beginning
summary
Airflow 3 is released (current 3.0.2) after ~8–9 months and a 20–50 person effort, claiming to fix long-standing pain points while preserving backward compatibility. It ships a HTTP-based task SDK that enables non-Python workers (Golang beta, Rust/TypeScript in progress), DAG and bundle versioning plus asset-first triggers and UI-controlled backfills, and an edge executor running on Raspberry Pi and Windows for distributed execution. Bosch reports year-long stable use of the edge pattern; security improves by routing task access through the API server instead of direct DB access.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:03What is Airflow?
- 2:27The value of Airflow
- 5:36Evolution of Airflow 3
- 8:29Reasons for Airflow 3
- 12:13Foundational properties
- 16:01Platform and integration
- 18:29Real-world use cases
- 20:13SDK and language support
- 22:56Assets and event scheduling
- 24:23Modern UI and monitoring
- 26:21DAG management and versioning
- 30:18Architecture and security
- 31:24Edge execution
- 34:28Future and closing
- 36:39Q&A session
related talks
42m`New` Workflow Orchestrator in town: "Apache Airflow 2.x"
26mGo with the Flow: Automating Your Workflows with Airflow
38mAirflow DAGs Made Easy: Introducing the Astro SDK
36mApache Airflow @ Umuzi.org
43mHelp! My data is late!
52mBetter ETL with Managed Airflow in ADF
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
56mWhy Airflow is a secret ingredient in MLOps