Scaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
summary
Shows how a production backend ingests OpenTelemetry at scale: an OpenTelemetry Collector based pipeline with a custom DXO2 exporter, stateful metric ingesters backed by separate metadata and time‑series stores, and a Kafka buffer with sticky partitioning to route source‑consistent traffic. Claims: handling 3 million metric points/sec, 250 million unique metric series/day and 0.5 million spans/sec. Covers the problem that synchronous APIs caused registration storms and the concrete fixes—Kafka buffering, sticky rebalancing, and a persistent cache for metric UIDs.
AI-written summary · about
chapters (8)
- 0:00Scaling OpenTelemetry workloads
- 4:41OpenTelemetry integration challenges
- 7:32Metric ingestion needs stateful processing
- 10:20Handling 4 million metrics per second with Kafka
- 14:18Kafka-based ingestion benefits
- 17:11Stateful metrics scaling
- 18:36OpenTelemetry community contributions
- 19:21Q&A
related talks
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...
34mReliable OpenTelemetry at Scale: No Queue, No Problem
49mAdding OpenTelemetry to Production Apps Lessons Learned
05mOpenTelemetry in production for a year plus: the good the bad the ugly
35mOpenTelemetry for Busy People
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
40mDeploying OpenTelemetry Organizationally: From Proof of Concept to In-Production at Scale
30mPanel: Telemetry That Matters