Too Big for DAG Factories?
summary
Covers using Apache Airflow to programmatically author, schedule and monitor large-scale data workflows (ETL, manufacturing scheduling, batch coordination). Shows patterns for dynamic DAG generation from JSON configs, scaling DAG factories to thousands of DAGs, CI/CD for releasing DAGs and infra, and running many tasks concurrently, including a live demo and a linked code repo.
AI-written summary · about
related talks
35mOrchestrating success
48mFlow Is in the Air: Best Practices with Apache Airflow
36mGetting Started with Apache Airflow - Building Your First Workflow
27mModern ETL-ing with Python and Airflow (and Spark)
38mUse Apache-Airflow to Build Data Workflows
22mData pipelines != workflows: orchestrating data with Dagster
38mLessons Learned from the Migration to Apache Airflow
36mElegant data pipelining with Apache Airflow