A Journey of Migrating Data Platform from Data Center to Cloud
summary
Migration moves a 19+ PB data lake with 19+ PB and 2 trillion messages/month from colo to public cloud, replacing Mesos-based runtime with cloud-native stacks and exploring Kubernetes, and cutting costs via elastic provisioning. The team ports ETL and orchestration into a modular platform centered on Goku flow engine, exposes unified SQL data access service and REST microservices, integrates Superset notebooks and H2O, and finds metadata migration and keeping query/format consistency are the hardest, six‑to‑eight month technical pushes.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:34Lei Gao Introduction
- 2:28Agenda
- 3:37Company Introduction
- 3:57Operational Data
- 5:36Data Platform
- 7:03Data Center Architecture
- 9:58Demo
- 15:47Challenges
- 17:14Data Match
- 19:10Data Microservice
- 21:05The Journey
- 22:21Data Access Services
- 24:45Data Engineers
- 25:52Customization
- 27:51Metadata
- 29:08Migration
related talks
01h 03mDiscovering the value of a cloud data platform
48mThe Journey of a strongly coupled Monolith into the Cloud Land
15mData strategy: Getting ready for Cloud migrations
27mModernize Your Data Warehouse
26mEffective Data Migration: Challenges & Success
34mFrom Data Centers to Cloud Native - Dave Zolotusky & James Wen, Spotify
45mHadoop in the Cloud: The Story of Migrating Hundreds of Petabytes
21mMigrate Data, Mesh in mind