Apache Airflow in Production - Bad vs Best Practices
summary
Recommends upgrading to modern Airflow (avoid brand‑new .0 releases; prefers 2.5.3 unless a feature forces 2.6), use Postgres not SQLite and front it with PG bouncer, and lock Python dependencies. Uses dynamic task mapping and shows Kubernetes vs Celery tradeoffs: Kubernetes gives per‑task isolation and resource limits, Celery gives always‑on workers; combine Celery+Kubernetes for heavy jobs. Insists on component log retention, Prometheus/Grafana monitoring, infra as code (Terraform/Ansible/Pulumi), CICD deployment patterns, and Kubernetes PodOperator for isolated task environments.
AI-written summary · about
chapters (11)
- 0:00Introduction to Airflow
- 4:57Choosing Airflow versions
- 6:23Airflow components explained
- 9:35Database and storage pitfalls
- 11:20Designing workflows properly
- 15:18Infrastructure automation
- 17:47CI/CD for DAGs
- 18:50Logging and monitoring
- 21:59Team collaboration issues
- 23:24Leveraging Airflow community
- 24:07Q&A session
related talks
32mWhat's coming in Apache Airflow 2.0
46mA Practical Introduction to Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
25mRunning Apache Airflow at Scale
01h 08mThe State of Apache Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
31mIntro to GKE setup of airflow with helm and terraform
36mElegant data pipelining with Apache Airflow