Lightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
summary
Distributed tracing costs explode at scale so the practical path is specific: mixed sampling fails because head and tail decisions across services break traces and create blame. Dropping attributes at the provider helps a little but can break the provider UI and still pays egress. A custom OpenTelemetry sampler that marks unhelpful middleware and format spans to skip plus an OpenTelemetry collector running tail sampling processors cuts payload and egress, empirically reducing spans by about 40% (87→53 example) while keeping useful error and latency traces.
AI-written summary · about
chapters (10)
- 0:00Distributed tracing costs
- 1:03Debugging in microservices
- 2:25Tracing bills grow with scale
- 3:30Mixed sampling experiment failure
- 4:56Filtering duplicate span attributes
- 6:00Tail sampling limitations
- 7:23OpenTelemetry custom sampling
- 8:50Custom samplers reduce spans
- 10:36OpenTelemetry tail sampling
- 11:59Summary
related talks
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
21mLightning Talk: Distributed Tracing Without the Pain!
33mReal-World Sampling – Lessons Learned...
27mTricked Out Traces
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
34mReduce Complexity with Distributing Tracing
09mGraduated Project Lightning Talk: Using Trace Data for Monitoring & Alerting of Appl...
24mUnlocking Advanced Processing Capabilities with OpenTelemetry