“Gluing it all together” aka building a platform around Prometheus
summary
Managed platform that provisions per-tenant Prometheus instances, a central remote storage (mimer), Envoy proxy, a custom operator and per-tenant Grafana, serving over 500 developers across 150 k8s clusters and around 70 tenants. Uses prometheus-operator, remote_write and Alertmanager; developers don’t run Prometheus but can run exporters. Major pain: remote_write causes replay delays after data‑center shutdowns and duplicate/dropped samples in mimer, scaling and upgrades require manual work and need autoscaling and better replay/ordering.
AI-written summary · about
chapters (19)
- 0:00Introduction and speaker intro
- 1:10Company environment and scale
- 2:39Early monitoring challenges
- 3:48Goals for the platform
- 5:37Target users and Kubernetes setup
- 7:04Service discovery and Prometheus Operator
- 8:31Local vs remote alerting
- 10:00Managed Prometheus and push Gateway
- 10:41Democratizing metrics with tenants
- 11:47Central configuration management
- 12:53Happy path for new tenants
- 13:36Platform architecture and components
- 15:04Platform scale and rule management
- 15:47Pitfalls with remote write
- 17:59Missing data and monitoring agents
- 19:03Scaling Prometheus instances
- 20:08User issues and rate limiting
- 20:51Future plans and improvements
- 21:39Q&A session
related talks
24mBanking Observability at Scale
35mDeclarative Multi-Cluster Monitoring with Prometheus
43mBuilding a Multi-tenant Monitoring System with Grafana, Prometheus and Cortex
24mCost-Efficient Multi-Cluster Monitoring with Prometheus, Grafana & Linkerd
02h 43mPrometheus from A to Z
28mFrom Manual to Managed: Prometheus Agent Deployment at Scale
24mPractical Kubernetes Monitoring with Prometheus
10mUpstream-First, High Scale Prometheus Ecosystem