Keynote: 10 Million Spans Per Second: Lessons From Scaling OpenTelemetry at Reddit
summary
Reddit moves 8–14 million spans per second with an OpenTelemetry pipeline through Kafka into ClickHouse and describes how sampling, batching, and ingestion tuning make that possible. Highlights include 10 million spans/sec bursts, building a custom Kafka consumer that batches into ClickHouse local tables using CH Go, and remote sampling that the observability team must own. A sudden sidecar caused site outage forces a fail‑open design and stricter sampling control; schema choices focus on fast trace-ID lookups over ad hoc high‑cardinality queries.
AI-written summary · about
chapters (9)
- 0:00Tracing platform at Reddit
- 1:02Tracing in distributed systems
- 3:14Migration to in-house observability
- 4:36ClickHouse and OpenTelemetry choice
- 6:19Scaling to 10 million spans per second
- 8:26Sidecar outage lessons
- 14:05Driving service adoption
- 16:10Optimizing ClickHouse queries
- 18:40Q&A
related talks
23mEnhancing Asynchronous Communication Observability with OpenTelemetry
41mScaling Telemetry Systems with Streaming
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
34mReliable OpenTelemetry at Scale: No Queue, No Problem
27mFast DIY O11y with OpenSrc Analytics & Visualization
06mPREVIEW: Real-time Stream Processing at Scale
30mThe Challenges of Managing Open Source Infrastructure at Bloomberg
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch