Building Composable OTel Pipelines: CI/CD, Testing, Team-First, and Sca...
summary
DoorDash describes a production OpenTelemetry pipeline that runs 100 TB/day of logs and handles 15M+ spans/sec using a mirrored collector-contrib repo, an OpenTelemetry Collector Builder that emits per-pipeline binaries, Helm values for per-gateway configs, and ~30 custom components (Kafka, S3 Parquet, ClickHouse exporters). Tests include in-memory ClickHouse unit tests and collector integration tests; deployment uses Kubernetes, Kafka, SQS and Cloudflare log ingestion with a log router for sampling, load-shedding at the edge, profiling and memory-limiter autoscaling.
AI-written summary · about
chapters (11)
- 0:00OpenTelemetry journey intro
- 1:47Fragmented telemetry problems
- 3:34Composable OTel pipeline blocks
- 5:02Testing pipelines
- 6:26Custom OTel collector images
- 7:31Custom OTel components
- 8:13Go-to logging pipeline
- 9:59Logging control plane
- 13:51Cloudflare edge logs
- 18:26Scale and learnings
- 20:12Future improvements
related talks
34mReliable OpenTelemetry at Scale: No Queue, No Problem
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
28mEnd-to-end CI/CD observability with open source
25mSponsored Session: Unifying Observability at DoorDash using Open Stand...
30mMaking your CI/CD Pipelines Speaking in Tongues with Open...
45mFrom Noise To Signal: Building Smart Observability Pipelines With OpenTelemetry
40mDeploying OpenTelemetry Organizationally: From Proof of Concept to In-Production at Scale
05mTelemetry Pipelines: Your Recipe for Success