Scaling Mimir: less is more
summary
Prometheus remote_write feeds Mimir distributors and ingesters with hash ring sharding; the original target is 1.5 million series per ingester and the cluster holds 200 million series which implied ~390 ingesters across replicas. Tests show consolidating shards into fewer larger ingesters lowers CPU and overall memory use (partly aided by a remote_write config change), cutting active CPUs from 3600 to 600 and improving query latency 400 ms to 39 ms.
AI-written summary · about
related talks
15mHow a Single API Endpoint Saved Us 3000 CPU
42mPrometheus Intro and Deep Dive
35mPushing the Limits of Prometheus at Etsy
31mLong-Term Metrics with M3 and Prometheus
39mHow To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations
11mAggregating, Alerting and Graphing on Millions of Prometheus T...
29mStreaming Recording Rules for Prometheus, Thanos, and Cortex Using...
26mPrometheus That Scales