Lightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
summary
Operating a metrics stack at 3.5 trillion samples/day (≈40M samples/sec, 2B active series) exposes failures not seen at smaller scale: Thanos Receivers return 5xx from unbounded queries, Prometheus remote_write lags and Store Gateway partial results when expensive queries starve ingestion, and multi‑tier OpenTelemetry collectors create duplicates and out‑of‑order rejects. Fixes include shared Helm charts to enforce golden configs, tenant-aware rate limiting middleware and query‑layer circuit breakers, consolidating into multi‑tenant clusters, enforcing read+write cardinality budgets and using cardinality leaderboards to drive optimization.
AI-written summary · about
chapters (8)
- 0:00Trillion-scale metrics platform
- 1:24Config as code and golden path
- 2:49Multi-tenancy shift to shared clusters
- 4:13Query contention starving ingestion
- 5:35Cardinality costs on read vs write
- 7:03OpenTelemetry collector pitfalls
- 8:28Key takeaways and lessons
- 9:25Q&A on query tools
related talks
17mLightning Talk: Avoiding Chaotic Dysfunction: The ‘Don’t Do This’ Guide T...
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
13mLightning Talk: Observability-First DevSecOps: Building Resilient Multi-Cloud P...
31mHandling Billions of Metrics with Prometheus and Thanos
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry
25mMulti-Cluster Observability: What you need to survive with a fleet
36mRunning the Observability As a Service For Your Teams With Thanos
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...