Tuning OTel Collector Performance Through Profiling
summary
Uses Go's built‑in profiler and the PPR extension (OpenTelemetry profiles extend PPR) to find real issues: a Prometheus receiver heap region grows from ~0.5 GB to ~2.5 GB because the cumulative to Delta processor keeps entries in a sync map (cardinality leakage) — mitigation is configuring cache eviction; Windows process scraping spends ~88% of CPU time calling CreateToolhelp32Snapshot for parent PIDs, switching to a single WMI query cuts a scrape from ~740 ms to ~220 ms and disabling parent PID drops it to ~90 ms (PR open).
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 1:00Understanding profiling
- 2:59Getting started with pprof
- 3:42Case study: Memory leakage
- 8:17Case study: CPU performance
- 13:00Conclusion and future outlook
related talks
27mWho Ate My Resources? Trace LLM Performance in Real Time With OTel
47mHow I Solved Production Issues with OpenTelemetry (And How You Can Too)
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
31mTroubleshooting and Monitoring Production Applications using OpenTelemetry
25mPerfect Match: Correlating Continuous Profiling with Distributed Tracing for Stronger Observability
28mControlling data overhead from the OpenTelmetry Collector
15mJourney from Fluent Bit, Fluentd and Prometheus to Open...
26mProfiling Mastery: Decode Your Data and Elevate Observability