How Thanos Almost Snapped $100,000 from Our Infra Budget
summary
Company migrates Prometheus to Thanos + S3 and deploys Fluent Bit for logs, then misconfigures retention and downsampling causing huge storage and egress churn. Thanos compactor behaviour forces keeping full blocks long enough to re-downsample, leading to repeated uploads across AZs; Fluent Bit retries on Kubernetes API failures amplify traffic through NAT gateways. Estimated runaway bill reaches $100,000 if scaled across 67 clusters; engineers build a custom Thanos compactor with flexible retention and tune Fluent Bit retries so the solution works and costs fall.
AI-written summary · about
chapters (12)
- 0:00Cost-saving observability challenges
- 1:25Long-term metrics solution needed
- 3:10Thanos as chosen solution
- 6:00Thanos implementation setup
- 8:08Early anomalies expectations
- 9:13Network topology S3 costs
- 10:38High cardinality metrics
- 12:02Thanos downsampling retention
- 14:52Fluent Bit API failure costs
- 17:42Near-disaster $100,000 calculation
- 18:23Lessons learned best practices
- 19:50Q&A
related talks
27mWe Stopped Thanos from Snapping $100K from Infra
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
26mPrometheus That Scales
15mScaling Observability Stack for AIOPS
54mThanos: Prometheus at Scale!
32mAdopting Thanos Gradually Across All of LastPass Infrastructures
31mHandling Billions of Metrics with Prometheus and Thanos
36mRunning the Observability As a Service For Your Teams With Thanos