Reindexing in record time (...)
summary
Shopify moves mapping changes from MySQL into Elasticsearch by spawning coordinator jobs per pod that create worker jobs per shop, running up to 30,000–50,000 concurrent workers across pods to reindex 800,000 merchants into new index versions. Jobs emit denormalized documents into a System5 queue then into Kafka aggregate for cross‑region durability, with iteration API slices (120s) to avoid blocking Resque. Large shops get split later, current worst cases cause a small latency spike when swapping aliases during the reindex under 24 hours process.
AI-written summary · about
chapters (5)
- 0:00Start
- 10:04Iteration Api
- 13:48Why We Use Kafka
- 15:38My Sequel Query Optimization
- 16:53Add Top Level Index Concurrency
related talks
28mEnhancing Elasticsearch Performance: Innovative Reindexing...
39mScaling Shopify's Multi-Tenant Architecture across Multiple Datacenters
38mScaling an online search engine to thousands of physical stores
39mZero-Downtime Rebalancing and Data Migration of a Mature Multi-Shard Platform
44mSurviving Elasticsearch
49mReplatforming Online Search for a Major High Street Retailer
42mOpenSearch Migrations: Modernize Without Worry
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond