Inside the Telemetry Data Plane: Constraints, Tradeoffs, and Scale
summary
Telemetry data planes confront three hard limits: memory, scheduling, and the throughput/latency budget. Fluent Bit-style agents use 2 MB soft chunk limits and in-memory vs filesystem buffering trade durability for IO and sync latency, while a single-threaded engine makes retries and blocking filters create CPU spikes and retry storms that can saturate backends. Practical patterns are per-destination isolation, explicit drop policies, chunk ownership tracing, and exposing metrics (Prometheus endpoint) to observe and debug production behavior.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:03Telemetry data plane
- 2:49Core constraints of pipelines
- 3:33Memory buffering trade-offs
- 9:17CPU bottlenecks
- 11:43Throughput vs latency budgets
- 16:40Scale trade-offs and reliability
- 22:25Survivable pipeline patterns
- 26:20Key takeaways and closing remarks
related talks
05mTelemetry Pipelines: Your Recipe for Success
35mHigh Throughput with Low Resource Usage: A Logging Journey
27mThe Signal in the Storm: Practical Strategies for Managing Telemetry Overload
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
35mRefining Systems Data without Losing Fidelity
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
08mStreamlining Telemetry Data