Behind the scenes of an ELK system
summary
Architecture that handles billions of logs a day from 15,000 devices uses Filebeat -> HAProxy (SSL terminate) -> Logstash receivers and separate Logstash processors, with RabbitMQ per-pipeline queues between them, and Elasticsearch hot/cold nodes behind an HA proxy and keepalived pair. Implementation uses 34 processor pipelines, ~15k persistent TCP connections, ~60 billion logs and 85–95 TB stored, with JVM heaps capped at 31 GB heap per data node; the setup is resilient and operational at scale.
AI-written summary · about
chapters (15)
- 0:00Session introduction
- 2:13Data management context
- 3:59Architecture considerations
- 8:55Comparing software solutions
- 10:37Agent and input configuration
- 15:40Logstash pipeline design
- 20:05Advanced architecture scaling
- 27:56Load balancing with HAProxy
- 41:55Scaling processor throughput
- 47:21Cluster high availability
- 53:57Mitigating single points
- 1:01:59System monitoring and stats
- 1:05:37Elasticsearch optimization
- 1:14:23Index strategy and access
- 1:18:57Final Q&A and wrap up
related talks
42mMake your logs work for you
40mA needle in a logstack
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
01h 00mApplication Logging & Logstash
46mBuilding a scalable logging platform
43mPractical Application Logging and Monitoring
41mScaling Observability: From Chaos to Clarity with Open Source Tools