You Know, for Search. Querying 24 Billion Records in 900ms
summary
Practical lessons from designing and running a 120-node Elasticsearch cluster (6 TB) and related Hadoop integration to serve very large search datasets (24 billion records). Covers cluster elasticity, AWS vs dedicated hardware trade-offs, SSD/EBS/instance-store I/O, EC2 instance sizing, Elasticsearch tuning and custom facet workarounds for OOM, maintaining responsiveness during heavy indexing, automated deployment and upgrades (e.g. Puppet), monitoring tools (Ganglia, Zabbix, Elasticsearch-Head, Bigdesk), cost comparisons, and using Hadoop/MapReduce/Hive/HDFS to populate and back up the search cluster.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 3:37Search technology evolution
- 8:07Elasticsearch setup
- 13:50Performance scaling
- 17:49SSD optimization
- 18:51Q&A
related talks
44mElasticsearch at Scale of Billions of Documents
38mScaling an online search engine to thousands of physical stores
33mMulti-Terabyte Sphinx HA Cluster
32mSide by Side with Elasticsearch & Solr part 2
35mHighly Available Search at Shopify
44mSearching through large graphs using Elasticsearch
43mCattle vs. Pets - A Cost-Effective Elasticsearch Architecture to Scale-Out...
43mMulti-tenancy for all workloads