Airflow: Save Tons of Money by Using Deferrable Operators
summary
Apache Airflow 2.2 adds deferrable (async) operators that suspend tasks on the worker, move polling to a central triggerer, and free worker slots; Astronomer ships many drop‑in async providers. Benchmarks show up to ~90% resource reduction when idle polling exceeds ~10 minutes, and overhead makes deferral pointless under ~20–30 seconds. Deferrable operators require an external unique job ID to track work, and trigger logs do not automatically appear in the web UI.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:45What is Airflow
- 2:43Why do we need variable operators
- 5:07Typical operators
- 9:18Main point
- 10:23Async operators
- 11:58Trigger
- 13:15Sync Operator
- 15:44Sync Operator vs NonSync Operator
- 16:47Trigger Baseline
- 19:55Trigger Example
- 24:59Available Operators
- 27:47Example
related talks
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
22mOrchestrate your data pipelines using Apache Airflow
36mApache Airflow in Production - Bad vs Best Practices
42m`New` Workflow Orchestrator in town: "Apache Airflow 2.x"
12mHow Affirm Uses Temporal as a Batch Data Plane
26mGo with the Flow: Automating Your Workflows with Airflow
38mProcrastination as a Service: Caching and Queueing
35mBecome a Conductor with Apache Airflow