Lessons Learned from the Migration to Apache Airflow

on YouTube ↗38m

" summary

Migrates a Hadoop stack to BigQuery and Apache Airflow, processing hundreds of terabytes, ~80 billion impressions and ~500 million clicks that translate to about $800M in ecommerce value; orchestration uses Airflow with Kubernetes + Celery executor, Apache Beam/Dataflow for ingestion, BigQuery for storage, Spark for heavy ML, and exports to S3/GCS and Elasticsearch. Claims doubled feature throughput after migration, enforces idempotent DAGs and integration tests (airflow test + CI), and stores persistent logs in S3/GCS to survive worker failures.

AI-written summary · about

" chapters (14)

1. 0:00Intro
2. 0:11Lessons learned from the migration to Apache Airflow
3. 0:49Agenda
4. 1:42Skimlinks: What we do
5. 4:00Why Airflow?
6. 8:05Data Architecture Overv
7. 10:48Airflow and Spark
8. 13:08DAG: Directed Acyclic Graph
9. 16:22Operator
10. 19:48Advanced Features
11. 23:18Sample code
12. 25:58Idempotent DAGS
13. 29:00Best practices: Docker and Kubernetes environments
14. 32:15Airflow: The Good, the Bad and the Ugly

" related talks

• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 32mBuilding Analytics Workflow using Airflow and SparkYohei Onishi · PyCon APAC 2019
• 25mRunning Apache Airflow at ScaleJean-Martin Archer, Michael Petro · PyData Global 2022
• 16mBridging Data Science and Data Infrastructure with Apache AirflowDaniel Imberman · PyCon US 2020
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 32mOrchestrating Your Data Pipelines with Apache AirflowRicardo Sueiras · GOTO Amsterdam 2022
• 36mElegant data pipelining with Apache AirflowBolke de Bruin · PyData Amsterdam 2018
• 48mFlow Is in the Air: Best Practices with Apache AirflowDominik Benz · PyCon.DE 2017