Ingesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
Talkat Observability Day Europe 2023 · on Tuesday, 2023-04-18· at 11:55· Hall 7, Room E | Ground Floor | Europe Complex·
summary
Architecture ingesting 6.5 TB/day of telemetry using OTLP and custom OpenTelemetry Collector binaries built with Collector Builder, exporting logs to AWS OpenSearch (about 4 TB/day), metrics to AMP (150M active time series) and traces to Honeycomb (2.15B events), yielding ~41% observability cost reduction. Key operational pieces are Argo CD deployments, Grafana for centralized dashboards and RBAC, sharded pipelines, per-index sampling extension, and a write‑ahead log to S3 with Lambda backfill for sink outages.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:04Agenda and problem
- 2:51Centralized observability challenges
- 5:42Solution architecture
- 7:28Diagnostics library design
- 11:45Deploying collectors
- 13:10Customizing collectors
- 15:20Visualization and governance
- 16:23Current data volumes
- 17:07Ensuring resilience
- 20:17Sampling and WAL strategies
- 22:46Alerting and incidents
- 23:27Migration steps
- 25:34Recap
- 26:38Q&A
related talks
37mObservability For You And Me With Opentelemetry
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
30mPanel: Telemetry That Matters
36mObservability For You and Me with OpenTelemetry
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...
49mAdding OpenTelemetry to Production Apps Lessons Learned
34mReliable OpenTelemetry at Scale: No Queue, No Problem
35mOpenTelemetry for Busy People