Enhancing Elasticsearch Performance: Innovative Reindexing...

on YouTube ↗28m

" summary

Shopify separates heavy reindex work onto a dedicated reindex node pool on GKE and drives autoscaling with KEDA with a Prometheus trigger that measures shards per node, so nodes scale up only for a reindex. The migration handles a ~100 TB orders index and yields a 40% faster reindex, a 98% drop in contention on real‑time nodes, and reduces real‑time pool CPU and memory by 58% and 15% respectively for a ~43% infrastructure cost saving.

AI-written summary · about

" chapters (8)

1. 0:00Shopify search intro
2. 3:28Elasticsearch at Shopify
3. 7:20Real-time vs reindexing
4. 15:18Shared compute problem
5. 17:23Separate workloads solution
6. 20:16KEDA autoscaling
7. 26:39Results and savings
8. 27:46Q&A

" related talks

• 32mScaling Shopify's Search: Enhancing Elasticsearch Resilience With Kubernetes and KE...Leila Vayghan · KubeCon + CloudNativeCon Europe 2025
• 20mReindexing in record time (...)Conor Landry · Berlin Buzzwords 2019
• 29mHow (not) to Scale Elasticsearch for Data Analytics!Aishwarya Sankaravadivel · Developer Summit 2020
• 38mScaling an online search engine to thousands of physical storesAline Paponaud · Berlin Buzzwords 2022
• 44mSurviving ElasticsearchVitaliy Kharytonskiy · Highload fwdays'18
• 49mReplatforming Online Search for a Major High Street RetailerStephen Tjasink, Sue Clutton · Agile on the Beach 2021
• 32mAutoscaling Elasticsearch for Logs on KubernetesCiprian Hacman, Radu Gheorghe · Berlin Buzzwords 2022
• 32mAutoscaling Elasticsearch for Logs on KubernetesCiprian Hacman, Radu Gheorghe · KubeCon + CloudNativeCon Europe 2022