From Sampling To Full Visibility: Scaling Tracing To Trillions of Spans
Talkat Observability Day 2025 · on Tuesday, 2025-04-01· at 12:55· Level 1 | Hall Entrance S10 | Room C·
summary
Claims a move from Zipkin→OTel migration and 100% sampling for critical flows while keeping costs low, validated on a workload of 3 trillion spans/day where 1% of traces contain most errors. Builds a columnar store with columnar storage + delta encoding, Bloom filters, partitioning and tiered hot/warm/cold storage, plus special shards for errors and slow spans to speed lookups. Reports 10–20x compression, another 2–3x from encoding, and 50–100x query improvement enabling full visibility without prohibitive storage costs.
AI-written summary · about
chapters (9)
- 0:00SNMP to full observability
- 1:50Debugging with tracing signals
- 5:23Happy traces mystery
- 7:10Tail sampling challenges
- 8:13Trace data explosion scale
- 9:18100% sampling techniques
- 12:48Error traces special shards
- 13:51Tiered storage for cost
- 14:55Key takeaways on tracing
related talks
29mWide Event Analytics
33mReal-World Sampling – Lessons Learned...
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
11mCognitive and Self-Adaptive System for Effective Distributed-Tracing...
24mCracking the Cost Code: A Cost Optimized Approach to Distributed...
26mDistributed Tracing: All the Warning Signs Were Out There!!!
29mIntroduction to Distributed Tracing