Elasticsearch at Scale of Billions of Documents
summary
Egnyte runs three independent Elasticsearch clusters on Google Compute Engine serving billions of documents and dozens of terabytes with 21 data nodes, three master/client nodes, and HAProxy in front. Migration from an old Lucene engine delivers the search box but exposes real issues: split‑brain unless min_master_nodes is set, long rolling‑restart recoveries on ES 1.3 (fixed in 1.6), fielddata OOMs from unbounded sorting/aggregations, mlock must be checked, and avoid G1 GC. Useful ops tools: HAProxy, Kopf HQ Bigdesk.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:46Agenda
- 1:52What is Elasticsearch
- 4:08Terminology
- 6:09Database Terminology
- 7:07Inverted Index
- 7:19Inverted Index Example
- 8:30Elastic Search Nodes
- 10:26Cluster Health
- 11:35Query Results
- 11:45Ignite
- 17:29Architecture
- 20:46Problems
- 22:31Split brain issue
- 25:10Consistency vs Availability
- 27:43Rolling Restart
- 31:55Memory Management
- 33:58Locking Memory
- 38:43Garbage Collector
related talks
19mYou Know, for Search. Querying 24 Billion Records in 900ms
29mHow (not) to Scale Elasticsearch for Data Analytics!
16mSo many results... Give them to me!
44mSurviving Elasticsearch
01h 01mBest Practices With Elasticsearch
24mOperating at scale with Elastic Search in production
35mElasticsearch - Beyond Full-text Search
16mTop Elasticsearch tips