Cattle vs. Pets - A Cost-Effective Elasticsearch Architecture to Scale-Out...
summary
Managing tens of petabytes and trillions of logs monthly, the architecture moves from single massive clusters to a “cluster of clusters” with a centralized management cluster, HTTP/TCP health checks (status served on port 9600) and a custom scoring service using the USE Method to route traffic away from stressed clusters. An auditing sidecar aggregates slow and security logs via UDP to identify and throttle bad queries. Outcomes: 57% lower compute cost, ~4x faster indexing, and migrations cut from 14 weeks to 2–5 days under a cattle rate zero model.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:20Talk agenda
- 4:11Aggregated logs for Workday
- 5:57Elasticsearch scaling problem
- 7:01Vertical vs horizontal scaling
- 9:11Multiple Elasticsearch clusters
- 12:01Ingestion path challenges
- 15:50Need for redesign
- 16:55Cattle vs pets
- 18:19Cattle architecture design
- 20:06Load balancer setup
- 24:43Architecture overview
- 27:55Query performance impact
- 29:46Indexing improvements
- 31:32Auditing service
- 35:02Dashboard insights
- 37:33Data cluster uniformity
- 39:18Architecture outcomes
- 41:06Migration efficiency
related talks
54m7 Petabytes of Logs in Elastic. How We Did It
57mBuilding a Scalable Analytical Solution Using IaaS and Elasticsearch
29mHow (not) to Scale Elasticsearch for Data Analytics!
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
48mBuilding Petabyte-Scale PostgreSQL Deployments
19mYou Know, for Search. Querying 24 Billion Records in 900ms
24mCracking the Cost Code: A Cost Optimized Approach to Distributed...
42mMake your logs work for you