Thanos Receiver Deep Dive
summary
Deep dive into a Thanos Receiver incident where remote_write from 10,000 hosts via Telegraph causes a surprising four-hour outage driven by a single “monster” query that triggers cross-tenant 409 loops; root causes include replication/quorum semantics with replication factor 2 and interaction with buffered Telegraph pipelines. Fixes that help include enabling receiver store.limits, switching to replication factor 3 and disabling dynamic hash ring scaling for stability, and using the out‑of‑order write window; the mitigations reduce recurrence though one timing behavior (old-data replay) remains unexplained.
AI-written summary · about
chapters (9)
- 0:00Thanos Receiver intro
- 2:10Receiver design and purpose
- 6:08Multi-tenancy and resilience
- 10:44Incident root cause
- 13:51Failed mitigation attempts
- 18:30Hypothesis on cause
- 19:11Mitigation steps
- 21:40Key takeaways
- 22:47Q&A
related talks
39mThanos Deep Dive: Inside a Distributed Monitoring System
25mWe're Still Down: A Metastable Failure Tale
37mWhen the Cure Is Worse than the Disease: Metastability in Recovery
29mHorrors of Distributed Systems
56mFrom Unreliable to Resilient: Mastering Distributed Systems
29mPost Mortems for 4 Years of Remote Execution
20mFrom timeout to sub-second: solving scale dependent deadlocks in distributed systems
26mBreaking Magical Barriers