Migrating Etsy infrastructure from On-premises to Google Cloud Platform
summary
Etsy migrates production to Google Cloud Platform and runs a parallel on‑prem fallback after an initial flip failed then succeeded; analytics migration follows next year. Infrastructure uses Kafka binlog readers to stream MySQL to Vertica, HDFS and BigQuery; Vertica stays for ad‑hoc OLAP while BigQuery handles other analytics. Key numbers: production cutover Aug 19, 500 TB analytics stack, 40 new 1 TB‑RAM servers cost $1M. Elastic verticals and shutting down compute reduce cloud costs.
AI-written summary · about
chapters (20)
- 0:00Intro
- 2:10Etsy Fun Facts
- 4:23Why Move to the Cloud?
- 8:41Cloud Migration Planning
- 10:21Elasticity
- 10:51Etsy Infrastructure
- 12:35Cloud Economics
- 13:38Production Data vs Analytics Data
- 15:03Comparing Enterprise vs Cloud OLAP
- 16:36Architecture for Elastic OLAP
- 18:04Relational Database Indexes
- 18:54Columnar Databases: Sorting and Encoding
- 20:25Example Query
- 21:00Relational Database Query Path
- 21:10Columnar Store Query Path
- 21:43OLAP Disadvantages
- 22:45Optimizing Server Deployment in Cloud
- 24:07Cloud Server Provisioning
- 25:25Rethinking Vertica Architecture in the Cloud
- 26:54Partition Cloud Apps into "Verticals"
related talks
46mHow Back Market Rebuilt Its Platform by Changing Cloud Provider
41mShopify's Move from the Data Centre to the Cloud
31mMigrating Legacy Infrastructure to Kubernetes: Olark’s Lessons From the Trenches
24mFrom Lift-and-Shift to K8s: A Two-Phase DevOps Modernization
19mBuilding an On-Premise Kubernetes...
37mBalancing Power and Pain: Moving a Startup From a PaaS to Kubernetes
31mChallenges of Migrating a High Traffic Mobile Game to Kubernetes
49mHardware to AWS, EC2 to EKS: Two Years, Two Epic Migrations