What's coming in Apache Airflow 2.0
summary
Apache Airflow 2.0 is not backwards compatible and moves to Python 3 only; a team generates about 130 operators with a code-generator that speeds productivity roughly tenfold and switches tests to pytest. Kubernetes-native executor proof-of-concepts fail because workers are stateful, so development pivots to a pub/sub native executor (horizontally autoscalable), an HTTP-based API for UI/CLI, stateless webserver images, better auth and monitoring, and operators moved to a new import path with backports for 1.10.
AI-written summary · about
chapters (7)
- 0:00Apache Airflow intro
- 8:59Why Airflow over others
- 15:03Airflow 1.x state
- 17:10Cloud-native Airflow 2.0
- 23:07Airflow executors & Kubernetes
- 29:10Airflow 2.0 improvements
- 31:17Workshop & closing
related talks
01h 08mThe State of Apache Airflow
36mApache Airflow in Production - Bad vs Best Practices
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
36mElegant data pipelining with Apache Airflow
25mRunning Apache Airflow at Scale
22mOrchestrate your data pipelines using Apache Airflow
06mComparing Argo Workflows and Apache Airflow at Scale
16mBridging Data Science and Data Infrastructure with Apache Airflow