99.99% of Your Traces Are (Probably) Trash
summary
Sampling reduces trace volume by design: use OpenTelemetry SDKs and the OpenTelemetry collector with head (probabilistic, rate-limit, adaptive) and tail (latency, error, attribute) policies, or combine them. Empirical claim: you can safely drop to 0.1% in some contexts; worked example shows 600,000 checkout traces generated, tail-only stores ~16,395 but combined 1% head + tail stores 600,000→164 traces while preserving nearly all errors and high-latency examples.
AI-written summary · about
chapters (11)
- 0:00Distributed tracing struggles
- 2:10Tracing adoption history
- 6:22Sampling crucial for cost
- 9:52Spans to traces in collector
- 12:41Head vs constant sampling
- 17:19Tail-based sampling policies
- 21:55Sampling strategies case study
- 28:36Dynamic sampling context
- 30:45Linking telemetry types
- 32:53Future of tracing
- 34:36Q&A
related talks
27mTricked Out Traces
33mReal-World Sampling – Lessons Learned...
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
35mRefining Systems Data without Losing Fidelity
06mObservability Diet: Your 5-Step Plan To Trim the Data Fat- 12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
39mWhy, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector
22mShift Into an Observability Mindset with OpenTelemetry