Telemetry Pipelines: Never Gonna Let You Down
summary
Explains concrete fixes for lost observability: use the OpenTelemetry Collector with an OTLP exporter to avoid routing collector-metrics through the pipeline, monitor the failed_send metric and container OOMs, and apply a memory_limiter processor plus Kubernetes HPA and external rate limits. Recommends queueing with retries, OTLP backpressure, tail sampling for spikes, and validating collector config before rollout; shows that these measures prevent cascading crashes and restore telemetry during backend outages.
AI-written summary · about
chapters (10)
- 0:00Telemetry pipelines intro
- 1:44What is a telemetry pipeline
- 4:59Reliability matters
- 6:05Pipeline telemetry
- 9:13Resource exhaustion
- 12:04Sending data failures
- 14:15Handling spikes and cascading failures
- 16:24Misconfiguration risks
- 18:13Availability zone failures
- 19:18Summary and best practices
related talks
25mDebugging the Pipeline: Observability for Your Observability with OTel + Prometheus
05mTelemetry Pipelines: Your Recipe for Success
11mLightning Talk: OTel Me How to Build a Data Pipeline for Observability
30mPanel: Telemetry That Matters
45mFrom Noise To Signal: Building Smart Observability Pipelines With OpenTelemetry
01h 03mGet a Grip on your Telemetry using the OpenTelemetry Collector
11mObservability 2.0 - Much more than just logs,metrics and traces
28mOpenTelemetry Is Not a Silver Bullet