Better Scalability & More Isolation? The Cortex “Shuffle Sharding” Story
summary
Cortex shards tenant traffic into deterministic, randomly chosen subclusters using a deterministic hash of tenant ID and then applies the usual quorum replication (write to three nodes, wait for two — replication factor 3). A simulator guided tuning to roughly 20–30k series per shard, which keeps node load variance small while cutting the chance a two-node failure affects many customers to about ≈2% tenants affected by two-node outage. Works with block storage/object stores, scales to hundreds of nodes, and limits noisy-neighbor and poison-request blast radius.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:55Overview
- 1:27Scaling Prometheus
- 6:03What is Cortex
- 7:32Timeline
- 9:42Scalability
- 13:27Shuffle Sharding
- 17:42Shuffle Shard Algorithm
- 22:35Results
- 23:47Recap
related talks
39mDeep Dive: Cortex
37mIntro: Cortex
20mCortex: Intro and Production Tips
32mSharding for mission-critical, internet-scale apps
28mCortex 101: Horizontally Scalable Long Term Storage for Prometheus
31mSharded & Distributed Are Not the Same:
52mTo Shard or Not to Shard
28mEfficiently distributing Postgres with Citus – How to choose the right shard key?