Introduction to orchestration ETL processes with Apache Airflow
summary
Demonstrates a Dockerized Airflow stack (webserver, scheduler, workers) orchestrating ETL between MS SQL Server and PostgreSQL using Python hooks and providers; builds and runs DAGs that use DAGs, Tasks, Variables, Params and XComs to extract, transform and load. Implements an initial load of 105 currencies into a Postgres data warehouse, a SQL sensor that detects Bitcoin insertion and triggers incremental loads, and a parameterized currency-rate loader with chunk size 20 — all examples run successfully on the provided environment.
AI-written summary · about
chapters (10)
- 0:00Apache Airflow and ETL intro
- 1:28Airflow basics and architecture
- 8:34Airflow workflows and tasks
- 19:22ETL tutorial with Docker
- 36:21Designing ETL with sample DBs
- 47:02Airflow web UI and first DAG
- 55:05Loading dimension tables
- 1:01:40Updating dimensions with sensors
- 1:09:35Parameterized ETL chunking
- 1:13:33Q&A
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
35mBecome a Conductor with Apache Airflow
46mGetting Started With Data Pipelines Using Apache Airflow
34mData Ingestion in Python using Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
16mBridging Data Science and Data Infrastructure with Apache Airflow
38mLessons Learned from the Migration to Apache Airflow