Go with the Flow: Automating Your Workflows with Airflow
summary
Covers practical Airflow use: DAGs, Operators, Hooks and concrete patterns like an S3 sensor that waits for a file, backfill replay to run historical intervals, and storing configs in a Postgres metadata DB. Shows Docker on Pivotal Cloud Foundry deployment, use of sqlite/moto for unit tests, migration away from SSIS, scheduler parallelism and restart behavior, sparse docs but active community, and common gotchas when templating and sharing operators.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:44Defining workflows
- 2:02Airflow basics
- 5:24Components and UI
- 8:29Development and best practices
- 10:48Operational demonstration
- 12:07Advanced scheduling and sensors
- 15:22Lessons and experience
- 17:48Technical deep dive
- 23:40Q&A and closing
related talks
35mBecome a Conductor with Apache Airflow
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
16mBridging Data Science and Data Infrastructure with Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
46mA Practical Introduction to Airflow
35mUsing Modern Application Principals to Automate Your Apache Airflow Data Pipelines