Journey from Fluent Bit, Fluentd and Prometheus to Open...
summary
Replaces Fluentd, Fluent Bit and Prometheus with OpenTelemetry Collector and a vendor-built OTEL distribution, unifying logs, metrics and traces. Fluentd cluster CPU drops from ~38 to ~13 cores and memory from ~220 GB to ~75 GB, later tuned to yield ~11–20× reductions; instance count falls from ~85 to ~20 and then ~11. Prometheus forwarding switches to the OTEL prometheus receiver and yields ~5× CPU and memory reduction. Some component bugs (file-lock receiver) exist but are fixed and the rollout works in production.
AI-written summary · about
chapters (10)
- 0:00Journey to OpenTelemetry
- 1:50Legacy logging challenges
- 3:35Why OpenTelemetry
- 5:21OpenTelemetry collector
- 7:08Adopting tracing first
- 8:35Performance gains with metadata
- 10:23Fluent Bit to OpenTelemetry logs
- 11:50Metrics with Prometheus
- 13:19OpenTelemetry issues
- 14:00Conclusion
related talks
36mFluent Bit: Logs, OpenMetrics, and OpenTelemetry all-in-one
57mTelemetry Showdown: Fluent Bit vs. OpenTelemetry Collector - A Comprehensive Benchmark Analysis
24mTelemetry Showdown: Fluent Bit Vs. OpenTelemetry Collector - A Comprehensive Benchma...
25mTelemetry Showdown: Fluent Bit Vs. OpenTelemetry Collector - a Comprehensive...
19mTelemetry Showdown: Fluent Bit vs. OpenTelemetry Collector - A Comprehensive Benchma...
33mObservability Projects Updates
49mObservability Day | Project Updates
28mOpenTelemetry developments in Prometheus over the last year