Highly Available Search at Shopify
summary
Shopify runs ~2 PB of search data across ~114 Elasticsearch clusters (largest ~120 nodes) and sustains a background indexing rate around ~90k/sec with reindex peaks near 400k/sec while serving millions of queries. Data is ingested via Kafka and consumed into geographically separated clusters to provide active-active via Kafka for regional failover. Kubernetes plus a custom controller enforces AZ rack awareness, does online volume expansion of disks, and throttles reindex consumers for real-time delay prioritization, keeping end-to-end freshness under ~1s.
AI-written summary · about
chapters (10)
- 0:00Introduction to Shopify search
- 2:52Shopify search platform team
- 5:40Failure domains in search
- 7:30Elasticsearch redundancy layers
- 10:40Regional failure protection
- 13:10Compute pressure challenges
- 17:21Storage growth handling
- 20:29Summary of failure mitigation
- 21:54Future challenges and scaling
- 22:37Q&A session
related talks
20mMigrating a Large Scale Search Dataset in Production in a Highly Available...
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...
19mYou Know, for Search. Querying 24 Billion Records in 900ms
40mElasticSearch: Big Data, Search and Analytics
44mElasticsearch at Scale of Billions of Documents
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
16mSponsored Keynote: OpenSearch at Atlassian Scale
43mMulti-tenancy for all workloads