How Pipedrive switched its observability stack to OpenTelemetry & LGTM
summary
Pipedrive migrates its observability to OpenTelemetry plus Grafana Mimir, Loki and Tempo, deploying OpenTelemetry collectors and custom wrapper libraries across 750 microservices and ~26,000 Kubernetes pods; timeline starts January 2022 and reaches ~200 live instrumented services within months. Recording rules reduced metrics by 97% (from ~1M to ~30K series) to make dashboards usable. Production footprint is ~1,000 containers, 360 CPUs, 1.3 TB RAM, ~300 TB storage and handles 2 billion logs per day; the stack is working in eight product regions.
AI-written summary · about
chapters (14)
- 0:00Introduction to Karl-Martin, Pipedrive, and Pipedrive's infrastructure
- 1:39Pipedrive's observability stack before LGTM and OpenTelemetry
- 3:32The challenges that Pipedrive faced with their previous stack
- 5:47Pipedrive's distributed tracing use cases
- 6:28What led Pipedrive to choosing OpenTelemetry and the LGTM Stack
- 7:14The timeline to get it all done
- 8:50Enabling OpenTelemetry
- 9:19Pipedrive's architecture
- 10:34Implementation challenges
- 11:46Recorded metrics vs. raw metrics
- 13:26Label cardinality: Understanding log and metric labels
- 14:00How we made the data useful (visualizations!)
- 18:09LGTM infrastructure by the numbers
- 19:19Pipedrive's future plans with Grafana
related talks
40mDeploying OpenTelemetry Organizationally: From Proof of Concept to In-Production at Scale
45mFrom Noise To Signal: Building Smart Observability Pipelines With OpenTelemetry
28mMonitoring Stack was yesterday – Observability Platform at scale!
21mBuilding Composable OTel Pipelines: CI/CD, Testing, Team-First, and Sca...
28mFrom Legacy Vendor Tooling To OTEL: Scaling Observability at MSCI With a Four-Person Team
28mEnd-to-end CI/CD observability with open source
05mOpenTelemetry in production for a year plus: the good the bad the ugly
01h 00mObservability made painless: Go, Otel & LGTM stack