Application Telemetry Through the Kernel With Linux user_events
summary
Linux user_events lets instrumented apps register tracepoints and write structured logs into per-CPU kernel/perf buffers so a listener can drain CPU-local rings without app-side queues. Microsoft builds an end-to-end pipeline: a user_events exporter for the OTel Rust SDK and a user_events receiver for the Otel-Arrow engine, pinned readers drain the same CPU. It enables on-the-fly log collection without restart, preserves locality and backpressure, but rings are bounded, require covering all CPUs to avoid silent loss, and need shared schema/decoding.
AI-written summary · about
chapters (14)
- 0:00Introduction and speaker background
- 1:25Problem with static telemetry costs
- 3:13Use case for dynamic log enabling
- 4:57What are Linux user events
- 6:43Three key ideas behind user events
- 9:50How user events work end-to-end
- 13:04Cost comparison with OTLP
- 14:27OpenTelemetry integration
- 16:12Differences from OTLP pipeline
- 17:57Kernel ring buffer limits
- 19:21Security and privacy considerations
- 20:22When to use user events
- 21:05Trade-offs and requirements
- 21:48Key takeaways and summary
