Scaling Observability: From Chaos to Clarity with Open Source Tools
summary
Observability traces jump from ~1GB to ~100GB in a month and later to ~1.5TB/day, breaking an Elasticsearch+Jaeger stack tested to 30k spans/s but failing at ~12k; logs pile up because Filebeat is starved, JVM heap fills with fixed bit set metadata (~500MB/index, ~21GB total), and on‑prem 5→13→42→67 data nodes still hit disk and network saturation. The team pivots to open source (Prometheus/Grafana, Filebeat/Logstash, Thanos, S3 snapshots), uses JBOD, closes old indices, throttles rebalancing, adds Prometheus HA and moves traffic off NAT (VPC peering, Graviton/RI) — these fixes stabilize throughput and cut cloud costs by crores; ClickHouse is evaluated as a JVM‑free alternative.
AI-written summary · about
chapters (26)
- 0:00Observability scaling journey
- 1:11Early architecture challenges
- 3:24Storage budget constraints
- 5:33Exponential growth broke predictions
- 6:58Log shipping delays in Kubernetes
- 8:22JBOD and horizontal scaling
- 9:28System overload signals
- 10:31Database snapshots vs Elasticsearch latency
- 11:55Hot-cold architecture blunder
- 13:40Elasticsearch memory limits
- 16:34Avoided sampling and backpressure
- 18:01Platform vs dev team blame
- 19:27Antivirus and hardware issues
- 21:13Large searches overwhelmed system
- 22:38Manual index management limits
- 24:02Key scaling lessons
- 25:48Diwali outage and Prometheus failure
- 26:52Troubleshooting to clarity
- 28:41Network saturation during rebalancing
- 31:06Reporting constraints and data limits
- 32:11Lift-and-shift to cloud costs
- 34:01Finance pushback on costs
- 35:27Cloud cost optimization
- 37:58Hot-cold storage lifecycle
- 39:07Learnings and unlearning scaling
- 40:30Future observability principles
related talks
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
25mOpen source observability for private cloud: mission impossible or not?
12mOpen-source observability is the way forward, and here’s why
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
31mMy Logs Cost More Than My Services
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
16mShrinking the Observability Bill