Astronomer's Journey from Cloud Lock-in to OSS Independence
summary
Migration narrative:Astronomer replaces AWS-managed pieces with open source on Apache Mesos/DCOS, swapping API Gateway+Lambda+Kinesis for a Go API, Kafka, Marathon and Apache Airflow; the Go rewrite yields a 10x improvement, pipelines scale from ~500 million to 3 billion events, and Airflow on Mesos runs millions of tasks a day with Marathon replacing Elastic Beanstalk. Built-in gaps: running Kafka in prod needs heavy tuning, S3 durability keeps some dependency, and most traffic still flows through Kinesis during the cutover.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:46Early priorities
- 1:11V1 of Astronomer
- 1:45Engineering obstacles
- 2:29Product obstacles
- 3:42Apache Airflow
- 4:37Why Apache Airflow
- 5:32mezzo scheduler
- 6:24Airflow
- 7:35Kinesis
- 8:31Road to Kafka
- 9:44Not all tools are created equal
- 10:20Exploration
- 10:31Terraform
- 11:18Infrastructure is Code
- 12:35Prometheus
- 13:01Monitoring
- 13:40Prometheus Monitoring
- 14:40Kairos DB
- 15:50Whats next
- 16:22Kafka Connect
- 17:04Amazon S3
- 18:05Kubernetes
- 18:34Questions
- 18:53Redundancy
related talks
22mFireside Chat with AWS
12mLeveraging Go for Efficient Infrastructure&Data Handling
31mA Journey of Migrating Data Platform from Data Center to Cloud
56mWhy Airflow is a secret ingredient in MLOps
32mMove to managed analytics
27mThe journey from legacy to cloud - native success
27mScalable by Design: How to Build an Elastic Cloud Infrastructure...
31mFrom Warehouse to Lakehouse: Data at Rest