Data governance with Airflow and ECS Anywhere
summary
An EU company acquires a US firm but US law requires US citizen data to be processed inside the US, so the solution extends the EU Airflow control plane to run containerized ETL on remote VMs via ECS Anywhere; jobs use the Airflow ECSOperator with custom instance attributes and task placement constraints to pin bonus or tax ETLs to hosts that hold the local data, ETL containers process CSVs and upload only result files back to EU S3, demo screenshots show runs with exit code 0.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:53Data governance basics
- 6:01Data residency challenges
- 8:34Cross-region ETL solutions
- 10:20Airflow and ECS Anywhere
- 15:42How they work together
- 17:49Demo walkthrough
- 29:14Solution recap and trade-offs
- 30:46Q&A
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
22mData Quality with Great Expectations and Airflow in a Reverse-ETL World
52mBetter ETL with Managed Airflow in ADF
38mLessons Learned from the Migration to Apache Airflow
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
32mBuilding Analytics Workflow using Airflow and Spark
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows