Apache Airflow in the Cloud: Programmatically orchestrating workloads w/ Py
summary
Practical tutorial that shows Airflow running real pipelines: code examples create dynamic DAG from CSV, use a GCS-to-BigQuery operator to load Avro files and verify a new BigQuery table, and demonstrate sensors that wait for a success file. Architecture notes cover sequential, local and Celery executors and tradeoffs. Demos run end-to-end on GCP and succeed; caveats include Python 3 quirks for some operators and older Airflow lacking RBAC.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 9:14Workflow management systems
- 16:50Apache Airflow benefits
- 26:28Airflow UI and concepts
- 35:23Airflow architecture
- 44:04Basic DAG tutorial
- 50:11Dynamic DAG tutorial
- 1:04:32XCOM and variables
- 1:08:48Google Cloud Platform
- 1:17:13GCP pipeline tutorial
- 1:23:23Summary
related talks
36mGetting Started with Apache Airflow - Building Your First Workflow
38mUse Apache-Airflow to Build Data Workflows
34mData Ingestion in Python using Apache Airflow
27mModern ETL-ing with Python and Airflow (and Spark)
32mBuilding Analytics Workflow using Airflow and Spark
22mAirflow Beyond the Cloud: Python Workflows at the Edge
38mAirflow DAGs Made Easy: Introducing the Astro SDK
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows