The Silent Symphony: Keeping Airflow's CI/CD and Dev Tools in Tune
summary
Apache Airflow runs 90 provider packages and depends on about 790 external projects, and the CI system (GitHub Actions) runs a full build in ~1h8m while selective checks can finish in ~5 minutes. The project enforces reproducible builds, runs the same dev environment locally and in CI via Breeze (Docker Compose wrapper), uses pre-commit hooks (132 hooks) to catch issues early, generates SBOMs, tests against lowest and latest dependency versions, and releases roughly every three months with ~2,900 contributors.
AI-written summary · about
chapters (11)
- 0:00Introduction to Airflow
- 1:39What is Apache Airflow
- 3:06The role of CI/CD in Airflow
- 8:05Developer tools for contributors
- 13:45Pre-commit automation benefits
- 17:45Local vs containerized dev environments
- 21:17Airflow's weekly activity stats
- 23:48Challenges in maintaining CI/CD
- 27:20The unsung heroes of CI/CD
- 28:45Call for action and thanks
- 30:10Airflow Summit promotion
related talks
28mManaging Python Dependencies at scale
40mContributor Experience matters
35mUsing Modern Application Principals to Automate Your Apache Airflow Data Pipelines
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
01h 08mThe State of Apache Airflow
21mAutomate your work
25mRunning Apache Airflow at Scale
41mContinuous Delivery Transformation: The New Way of Working