Shopify's journey from conventional to native histograms
summary
Shopify migrates its Thanos-based backend and dashboards from conventional to native histograms, ingesting both via Prometheus and a forked StatsD exporter with a protobuf native-histogram endpoint while contributing native-histogram support upstream. The change yields 10–12× cardinality reduction for many metrics (request-time examples drop from ~100M series to ~10M), speeds queries but raises decoding memory in queriers (~20–30%) and causes some SLO flapping from dynamic buckets; migration uses scripted Grafana/GitHub automation and per-team cutovers.
AI-written summary · about
chapters (9)
- 0:00Introduction to migration
- 1:28Conventional histograms explained
- 3:21Native histograms overview
- 5:47Challenges with classic histograms
- 8:02Migration to native histograms
- 10:10Dual ingestion setup
- 13:26Automated migration process
- 16:38Lessons learned from migration
- 19:29Q&A session
related talks
23mBeyond the Ephemeral: Mastering Serverless Metrics at Scale With Shopify
42mEvolution of Observability Tools at Pinterest
21mPrometheus Native Histograms in Production
04mLightning Talk: Prometheus Native Histograms Are Now a Stable Feature
35mDoorDash’s Journey From StatsD To Prometheus With 10 Million Metrics/Second
36mHow to Use Prometheus's Native Histograms
29mA Practical Blueprint for Evolving Observability Schemas
25mNative Histograms in Prometheus