The Art and Craft of No-touch Instrumentation
Talkat Observability Day 2025 · on Tuesday, 2025-04-01· at 15:20· Level 1 | Hall Entrance S10 | Room B·
summary
Automatic OpenTelemetry agents get injected into containers using LD_PRELOAD injection and an init container delivered by a mutating admission webhook, avoiding eBPF or extra kernel permissions. Demo shows Java, Node and .NET runtimes auto-loading agents from an ephemeral volume; injector built in Zig reads ELF to choose musl vs glibc artefacts. Caveats: fails for compiled Go/Rust/C++ and explodes if an agent file is missing, but the operator is running this in production today.
AI-written summary · about
chapters (14)
- 0:00No-touch instrumentation intro
- 1:27Why no-touch instrumentation matters
- 2:12What is instrumentation
- 4:41State-of-the-art in no-touch
- 7:28No-touch with Kubernetes
- 9:17Environment variables control
- 10:20Mutating webhooks for pods
- 12:06Dynamic linking basics
- 15:20Hijacking env variable lookups
- 17:07Demo of automatic injection
- 19:17OpenTelemetry operator challenges
- 21:08Musl vs glibc in containers
- 23:20ELF metadata and injector dev
- 24:24Key takeaways and future
related talks
39mAuto-Instrumenting Go Libraries for Tracing with eBPF and OpenTelemetry
45mSimplifying Instrumentation in complex systems at scale with Open Telemetry
25mWhere’s the Auto in Auto-Instrumentation? A Look at Current Automation Strategies with OTel
33mInstrument Go Without Changing a Single Line
30mObserving Python applications with OpenTelemetry
42mObservable Applications Using OpenTelemetry
23mOpenTelemetry Auto Instrumentation for Effortless Observability
40mInstalling OpenTelemetry, Today and Tomorrow