Scaling an online search engine to thousands of physical stores
summary
Describes building an Elasticsearch platform that indexes one index per physical store to keep local offers independent, accepts a marketplace feed of millions of SKUs and a warehouse of ~500k–1M items, and preprocesses data into a common schema so results can be merged. Runs thousands of store indices across about five clusters with a multi-cluster router, experiments with Kubernetes runners for heavy preprocessing, targets 0.5s search latency for the search component, and reports Black Friday scaling succeeded with this architecture.
AI-written summary · about
chapters (5)
- 0:00Start
- 3:54Real-world data
- 4:51Search masks complexity
- 19:15Example with Elasticsearch
- 23:17Monitoring
related talks
49mReplatforming Online Search for a Major High Street Retailer
19mEnhance Search Performance At Scale with
24mOperating at scale with Elastic Search in production
19mYou Know, for Search. Querying 24 Billion Records in 900ms
37mHow we solved for Aggregation, Search and Time Series Big Data usecases using Elasticsearch Database
28mEnhancing Elasticsearch Performance: Innovative Reindexing...
29mHow (not) to Scale Elasticsearch for Data Analytics!
44mElasticsearch at Scale of Billions of Documents