With OpenTelemetry, ComplyAdvantage overhauled its observability (twice)
summary
ComplyAdvantage migrates twice: from on‑prem Grafana OSS to Datadog, then to Grafana Cloud, enabled by heavy adoption of OpenTelemetry and an OpenTelemetry Collector sidecar pattern. The infra has 41 Kubernetes clusters and ~82 Prometheus replicas; span volume forces sampling changes from probabilistic sampling to a two‑layer gateway using a load‑balancing exporter and extracted span metrics to make deterministic tail sampling decisions. Grafana Agent replaces proprietary log agents, Prometheus runs in agent mode, and the new setup reduces noise and surfaces business‑facing metrics effectively.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:49Why Migrate? (Inspired by Google SRE Book)
- 3:45ComplyAdvantage Infrastructure
- 7:25OpenTelemetry Has Entered the Chat
- 10:31Distributed Tracing
- 11:35Observability Infrastructure
- 14:52Sampling - do you really need all that data?
- 16:44Timeline of the first migration
- 17:39Why we decided to migrate a second time
- 19:00The second migration, this time with Grafana
- 21:09Telling stories with data
- 26:00Timeline of second migration
- 26:18Lessons learned and reflection
related talks
05mOpenTelemetry in production for a year plus: the good the bad the ugly
46mObservability Ecosystem in Kubernetes
36mPanel: OpenTelemetry...
27mFrom Zero Visibility to Full Observability with Grafana
25mGrafana as Code-An Open Source Observability Solution
35mOpenTelemetry for Busy People
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
50mOpen Telemetry: The Rise of Open Standards in Observability