4 tricks that help you leverage your Airflow pipelines
summary
Four concrete recipes: use a CSV-based config to autogenerate templated DAGs so one line adds a new entity and separate start_date/catchup avoids backfill noise; use Google Sheets configs so non-devs create DAGs but add input validation and async polling to avoid scheduler timeouts; enumerate Airflow Connections at schedule time to autogenerate DAGs for multiple databases; make DAGs event-triggered (schedule_interval=None) and invoke them from Cloud Functions, accepting the external-service dependency.
AI-written summary · about
chapters (7)
- 0:00Introduction and speaker background
- 1:14What is Apache Airflow?
- 2:20Templated pipelines for entities
- 9:50EO delegation with Google Sheets
- 13:05Laser adaptation with connections
- 15:15Event-driven Airflow pipelines
- 18:12Q&A and closing remarks
related talks
26mGo with the Flow: Automating Your Workflows with Airflow
22mOrchestrate your data pipelines using Apache Airflow
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
52mBetter ETL with Managed Airflow in ADF
45mDefining data pipelines
18mHow we deleted 10,000 lines of code and got control of our Airflow DAGs
38mToo Big for DAG Factories?