Tricked Out Traces
summary
Sampling choices determine what traces survive at scale: using OpenTelemetry Collector with the spanmetrics connector preserves 100% of request metrics before sampling; doing head sampling naively (example 25%×10%×10% → 0.25% of payment traces, or the Katie/Mike 30%×20% → 6%) loses critical signals. Consistent probabilistic sampling (available in Java and Go) fixes cross-service ratios to the min (yielding ~20% in the example). Collector tail sampling supports rule sets (six policies tested), caches reduce CPU/memory, rule count adds only slight memory cost, and removing cumulative-delta metric conversion prevents apparent memory growth; enable collector metrics and use one enrich-then-loadbalance architecture with traceID affinity.
AI-written summary · about
chapters (14)
- 0:00Sampling in distributed tracing
- 1:47Why sampling matters
- 3:10Distributed traces basics
- 4:16Why sample traces
- 6:46Fixed ratio sampling challenges
- 7:52Spanmetrics for accurate metrics
- 8:56Head sampling in OpenTelemetry
- 11:06Probabilistic sampling explained
- 13:40Collector sampling methods
- 14:45Tail sampling alternative
- 16:29Tail sampling architecture
- 17:30Tail sampling performance
- 20:44Key takeaways
- 22:29Q&A
related talks
39mWhy, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector
33mReal-World Sampling – Lessons Learned...
35m99.99% of Your Traces Are (Probably) Trash
23mControlling Data Overhead with the OpenTelemetry Collector
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?- 12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
33mBalancing Cost and Quality in OpenTelemetry: An Evaluation of Sampling Policies