Taming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...
summary
Bloomberg builds a managed multi-region Karmada control plane stretched across multiple host clusters for HA and disaster recovery, using per-cluster Prometheus servers sharded and federated into Thanos for global query. Monitoring uses monitor nodes with fake kubelets and an Nginx metrics proxy to scrape protected API metrics, enriched with labels for tenancy and instance. SLOs are auto-generated with Sloth and defended by multi-window burn‑rate alerts. Karmada + Thanos + Prometheus 700 clusters 400 million time series
AI-written summary · about
chapters (10)
- 0:00Multi-cluster observability intro
- 1:23Multi-cluster orchestration at Bloomberg
- 3:49Managed Karmada platform
- 6:43Single-cluster observability
- 11:37Control plane metrics and dashboards
- 15:37Multi-cluster telemetry challenges
- 18:05Scaling metrics with Thanos
- 19:09Reliability with SLOs
- 23:01Karmada open source contributions
- 24:06Q&A
related talks
29mMonitoring multiple Kubernetes Clusters with Thanos
25mMulti-Cluster Observability: What you need to survive with a fleet
37mHow Bloomberg Creates a Resilient Data Analytics Platform Using Karmada
40mBuilding a Scalable Observability Platform for Edge Compute Nodes
37mAutoscaling Multi-Cluster Observability with Thanos and Linkerd
31mHandling Billions of Metrics with Prometheus and Thanos
26mAgile Monitoring At Scale With Thanos
25mMulti-cluster Orchestration System: Karmada Updates and Use Cases