Scaling Thanos at Reddit
recording made private on YouTubehttps://youtu.be/c18RGbAxCfI
summary
Reddit runs thousands of Prometheus instances managed by a custom Kubernetes operator, sharding by namespace and shipping data via sidecars into Thanos with S3 long‑term storage, holding about 2 petabytes and sustaining 45 million samples per second across 600 million active series. Autoscaling uses VPA but faces OOMs from cardinality spikes so scrape/sample limits, go mem limits, a label‑enforcer proxy for Thanos Query, and a centralized alert‑routing service handle stability and routing.
AI-written summary · about
related talks
26mPrometheus That Scales
30mIntro to Thanos: Scale Your Prometheus Monitoring With Ease
21mThanos: Easier Than Ever to Scale Prometheus and Make It Highly Ava...
37mThanos: Prometheus in HA Mode with Persistent Storage
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
29mMonitoring multiple Kubernetes Clusters with Thanos
26mAgile Monitoring At Scale With Thanos
25mMulti-Cluster Observability: What you need to survive with a fleet