How a Single API Endpoint Saved Us 3000 CPU
summary
Grafana Mimir ingestors on Azure spot nodes suddenly double their held series during rollouts because consistent-hash ownership moves when pods leave the ring, spiking CPU and breaking scheduling. Team reverse-engineers ingestor behavior, adds an API endpoint to toggle “leave ring on shutdown,” and runs a no-terminate daemonset that calls it on eviction notices so ingestors stay in the ring during rollouts but leave on preemption. Result: about 3000 CPU saved (~$14k/month).
AI-written summary · about
chapters (9)
- 0:00Observability platform intro
- 1:21Mimir and ingestor
- 3:29CPU spikes in rollouts
- 5:14Root cause of CPU spikes
- 8:45Metric series label issues
- 9:08Consistent hashing in rollouts
- 11:17Solutions brainstorm
- 13:23API endpoint fix
- 14:05Results and savings
related talks
06mScaling Mimir: less is more
41mScaling Observability: From Chaos to Clarity with Open Source Tools
29mStreaming Recording Rules for Prometheus, Thanos, and Cortex Using...
15mScaling Observability Stack for AIOPS
30mCortex: Insights, U...
26mPrometheus That Scales
39mHow we isolate streaming ingest from search using RocksDB
27mIngestify: Rethinking Ingestion for Complex Data