Why, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector
summary
OpenTelemetry Collector performs tail-based sampling by buffering full traces and applying policies in a tail sampling processor so a 20-call load generator that produces one error per run yields 20 traces with the default, ~60 after three runs, ~25% trace_id_ratio sampling gives ~7 new traces from 60, and a tail config with a status_code policy returns only error traces while adding a probabilistic policy returns errors plus a random sample. Config knobs include decision_weight, num_traces, expected_new_traces_per_sec. Limitations: memory and CPU for buffering, all spans must reach the same collector, need agent/collector topology and possibly the load balancing exporter.
AI-written summary · about
chapters (15)
- 0:00Start
- 2:18INTRODUCTION
- 2:31What is Open Telemetry?
- 3:14What is the Collector?
- 3:41What is distributed tracing?
- 6:26SAMPLING OVERVIEW
- 9:09Head-based sampling
- 13:26Tail-based sampling
- 14:11Tail sampling processor
- 16:15Demo context
- 16:56Sampling scenarios
- 28:00Demo reflection
- 28:50CONCERNS & LIMITATIONS
- 29:07General concerns
- 31:07Open Telemetry limitations
related talks
27mTricked Out Traces
33mReal-World Sampling – Lessons Learned...
23mControlling Data Overhead with the OpenTelemetry Collector
25mOpenTelemetry Collector Deployment Patterns
28mControlling data overhead from the OpenTelmetry Collector
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
33mBalancing Cost and Quality in OpenTelemetry: An Evaluation of Sampling Policies
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?