How we deleted 10,000 lines of code and got control of our Airflow DAGs
summary
Airflow pipeline for site classification shrinks from 100+ files to ~25 specific files plus ~25 in a reusable DSdag framework, removing roughly 10,000 lines over two to three months. A config-driven orchestrator replaces duplicate orchestrator files for daily, monthly and backfill runs; seven internal DAGs become instances of one class with one-line DAG creation and auto-generated docs. Runtime changes: output-ready sensor lets orchestrator proceed earlier so morning results arrive faster, local testing prefixes avoid clobbering prod tables, and the framework is already reused across projects.
AI-written summary · about
chapters (15)
- 0:00Introduction and context
- 1:58Inheriting technical debt
- 3:00Guiding principles for refactoring
- 3:21Tackling duplicate code
- 4:24Unifying orchestrators
- 5:48Standardizing internal DAGs
- 7:54Building the DS DAG framework
- 9:19Results and benefits
- 9:42Enabling ad hoc runs
- 10:45Speeding up pipelines
- 11:49Improving local testing
- 12:52Automating documentation
- 14:59Extending framework to other projects
- 16:02Summary and conclusion
- 16:44Q&A
related talks
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
26mModern Data Orchestrators
38mToo Big for DAG Factories?
22mOrchestrate your data pipelines using Apache Airflow
19m4 tricks that help you leverage your Airflow pipelines
45mDefining data pipelines
26mGo with the Flow: Automating Your Workflows with Airflow
01h 59mIntroduction to orchestration ETL processes with Apache Airflow