Lightning Talk: OTel Me How to Build a Data Pipeline for Observability
summary
Observability is framed as a data problem and the recommended approach builds a centralized pipeline using the OpenTelemetry Collector with receivers for Prometheus, cloud providers, Kafka (admin API, JMX, or Prometheus exporter), processors to inject context (the kubernetes attributes processor), filter data, and apply tail-based sampling to retain error or high-latency traces. Export uses OTLP to one or multiple backends. Operational notes: the collector needs careful scaling, load balancing, self‑monitoring, and some components are still of mixed stability.
AI-written summary · about
chapters (13)
- 0:00Speakers and talk intro
- 0:45Scaling observability data challenges
- 1:28Optimizing ROI in observability
- 2:50OpenTelemetry Collector architecture
- 3:51Integrating existing tools
- 4:54Kafka metrics with OpenTelemetry
- 6:18Customizing OpenTelemetry
- 6:39Kubernetes attributes for data
- 7:42Controlling data volume
- 9:30Exporting to backends
- 10:14Building data pipelines
- 10:34Collector challenges
- 11:15Closing and community
related talks
25mDebugging the Pipeline: Observability for Your Observability with OTel + Prometheus
37mObservability For You And Me With Opentelemetry
50mGuide to Observability with OpenTelemetry
36mObservability For You and Me with OpenTelemetry
18mOpenTelemetry and Observability: What, Why, and Why Now?
20mTelemetry Pipelines: Never Gonna Let You Down
45mFrom Noise To Signal: Building Smart Observability Pipelines With OpenTelemetry
13mManaging Observability Data at the Edge with the OpenTelemetry Collector and OTTL