An Elastic Search Cluster Named George Armstron Custer
summary
Elasticsearch used for centralized logs repeatedly causes whole-cluster failures so a runbook job called The Hammer automates restarts but does not fix root causes. A custom monitoring pipeline polls Elasticsearch API endpoints (nodes cluster health) and runs side-by-side comparisons between the old cluster and a new cluster named Winston. JVM metrics show a missing GC sawtooth and high JVM heap utilization, so alerts pull misbehaving nodes and either add capacity or retire indices. External checks via monitoring the API through the load balancer prevent blind spots and the automation lives at bitly/custerscluster.
AI-written summary · about
chapters (7)
- 0:00Blue Square vmware
- 1:45Failing servers crash entire cluster Servers unresponsive or inaccessible Cluster restarts take time
- 2:15Using Monitoring Endpoints
- 3:15Predict node failure by monitoring JVM Heap Remove node to prevent complete cluster failure
- 4:15Elasticsearch monitoring is internal Doesn't work when cluster is down Monitor externally via load balancer
- 4:30Teamwork is critical Ideas, tools, support are helpful
- 5:00Custer's Next Stand? How Trax monitors now Automatic monitoring in a dynamic environment
related talks
58mTroubleshooting your Elasticsearch cluster like a Support Engineer
40mMaking your elastic cluster perform
44mElasticsearch at Scale of Billions of Documents
14mOperatoration Testing of Cluster
45mMonitoring Your OpenSearch Cluster Effectively
54mScale Your Elasticsearch Cluster
01h 00mElasticsearch: you know, for more than search
41mRule #2: Double Tap. An Elasticsearch Journey of Resiliency