Lessons Learned from the GCP Data Migration
summary
PAYBACK moves terabytes into BigQuery and Cloud Storage using Cloud Dataflow (Apache Beam) with on‑the‑fly pseudonymization via Cloud DLP tokenization, then loads with BigQuery and MERGE INTO to upsert. A commercial tool called Gluent vanished mid‑project so the team rebuilds a native GCP pipeline, uses Cloud Run for reconciliation (DataDiff) and Cloud Composer for orchestration, hits and resolves DLP quota and billing surprises (logical vs physical bytes), and validates row‑level parity so the migration actually completes and works.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:06Payback overview
- 10:22Migration strategies
- 13:29Data mirroring requirements
- 16:13GCP architecture
- 21:54Configuration best practices
- 24:41Apache Beam insights
- 27:10BigQuery considerations
- 29:55Cost optimization tips
- 31:27Airflow best practices
- 35:20Data validation methods
- 37:03Infrastructure tips
- 39:34Key takeaways
related talks
36mMigrating Our Data Warehouse to the Cloud
47mData Warehousing Migrations: Lessons from Home Depot
37mBuilding a Data Platform from scratch
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
25mData Demons: Unleashing Chaos or Driving Performance?
08mCloud-Based Data Integration is Transforming Enterprise Data
27mNotorious B.I.G. Data: Wins and fails when building a big data platform in the cloud
33mManaging IoT Storage with Google's Cloud Platform