Sell cron, buy Airflow: Modern data pipelines in finance
summary
Quantopian replaces brittle crontabs with Apache Airflow pipelines in less than six months, turning scattered scheduling logic into DAGs, checkpointed S3 artifacts, and observable runs with PagerDuty and Slack alerts. A summer intern ports the flaky file mover and lands 70 PRs, and a new alpha-model pipeline runs on Airflow + Kubernetes with containerized steps. Nightly ETL, multi‑market calendars, and recoverable retries work in production and dramatically cut manual pages and operational friction.
AI-written summary · about
chapters (10)
- 0:00Modern data pipelines
- 7:23Cron-based ETL problems
- 12:37Business goals for modernization
- 17:36Why Airflow was chosen
- 21:51When to use Airflow
- 25:44Migrating to Airflow
- 27:53Airflow and Kubernetes benefits
- 30:24Future Airflow improvements
- 33:56Advanced Airflow use cases
- 34:48Q&A
related talks
22mOrchestrate your data pipelines using Apache Airflow
38mUse Apache-Airflow to Build Data Workflows
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
32mOrchestrating Your Data Pipelines with Apache Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
36mElegant data pipelining with Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
01h 59mIntroduction to orchestration ETL processes with Apache Airflow