We Stopped Thanos from Snapping $100K from Infra
summary
A chain of concrete misconfigurations drives the story: a missing S3 VPC endpoint forces all Thanos sidecar uploads over the public internet, exploding egress costs; Thanos Compactor default retention/downsampling timings cause full-resolution blocks to be re-ingested and stored longer than expected; high cardinality from dynamic labels multiplies samples; Fluentbit fails to call the Kubernetes API because of a bad subpath mount and stale certs, producing anonymous logs and memory spikes. The fixes are a custom compactor with configurable resolutions and retention constraints, tightened developer permissions and review gates, and VPC endpoint and alerting changes that prevent the multi-cluster ~$100K bill.
AI-written summary · about
chapters (11)
- 0:00Introduction to talk
- 1:25Why we needed Thanos
- 4:13Thanos setup and expectations
- 6:20Initial setup failures
- 7:24Network topology issues
- 9:31Downsampling misconfiguration
- 14:48High cardinality metrics
- 16:55Fluent Bit log aggregation failure
- 20:05Cost of misconfigurations
- 22:15Final lessons learned
- 23:40Q&A session
related talks
27mHow Thanos Almost Snapped $100,000 from Our Infra Budget
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
15mScaling Observability Stack for AIOPS
33mStory of Correlation: Integrating Thanos Metrics with Observabi...
36mRunning the Observability As a Service For Your Teams With Thanos
26mPrometheus That Scales
05mTroubleshoot Compactor Backlog with Ease
32mAdopting Thanos Gradually Across All of LastPass Infrastructures