Migrating a Large Scale Search Dataset in Production in a Highly Available...
summary
Shopify migrates Elasticsearch data across GCP regions while running clusters on GKE, using CFA for the indexing pipeline and SQL as the primary store. The problem: a single Shopify core cluster holds ~400+ TB and ~400 billion documents, with indexing peaks of 990,000 docs/s realtime and 500,000 docs/s reindex. The chosen solution enables cross-jurisdictional writes, takes snapshots, then performs automated per-shop backfills to avoid search downtime at the cost of temporary extra infrastructure and operational complexity.
AI-written summary · about
chapters (12)
- 0:00Introduction to Shopify search
- 1:20Shopify platform overview
- 3:11Core search technologies
- 5:08Data ingestion pipelines
- 6:47High availability strategies
- 8:02Indexing and system scale
- 9:39Global expansion challenges
- 10:57Data migration hurdles
- 12:57Evaluating migration solutions
- 15:38Analyzing alternative approaches
- 17:05Cross-jurisdictional rights
- 18:41Automated backfills and future
related talks
35mHighly Available Search at Shopify
41mShopify's Move from the Data Centre to the Cloud
36mHow we finally migrated an eCommerce-Platform to GCP
34mArchitecting Solr indexing pipelines in Google Cloud Platform
11mPractical Migration From Distributed OpenSearch Environments To Enterprise-wide SIEM Platform
46mHow Back Market Rebuilt Its Platform by Changing Cloud Provider
40mElasticSearch: Big Data, Search and Analytics
50mOur experience at Etsy com with building “Big Data” analytic capabilitie