Distributed Tracing - All The Warning Signs Were Out There!
summary
eBay runs tracing at ~400 billion spans/day (about 5M spans/sec), 3,000 CPUs and ~200 TB for 14‑day retention, using OpenTelemetry and ClickHouse sharded into clickhouse rings; a managed framework covers 80–90% of apps with monthly upgrades but misused threadpools break context. Cost drives a 2% sampling baseline, tail sampling proves brittle across regions so a pipeline uses exemplars, delayed tail decisions (10–15 min), and local collectors; result: useful but fragile and still evolving.
AI-written summary · about
chapters (11)
- 0:00Introduction to tracing
- 1:45Defining the core problem
- 3:31Strategy and implementation
- 7:44Platform architecture
- 10:05Optimizing for cost
- 12:06Tail sampling and scaling
- 14:18Real-world challenges
- 18:03Solving performance issues
- 19:51Refining sampling approaches
- 23:45Machine-centric observability
- 26:58Future outlook and Q&A
related talks
26mDistributed Tracing: All the Warning Signs Were Out There!!!
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
41mTracing Real-Time Distributed Systems
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
50mDistributed Tracing in the Wild
22mDistributed Tracing
20mTTChat’s Story: Connect Metrics, Logs and Traces with EBPF
32mDistributed Tracing in Action: Our Journey with OpenTelemetry