How To Think About Instrumentation Overhead
summary
Instrumentation consumes CPU, memory, startup time and bandwidth and that cost is often roughly constant per span so latency percentages are lies: percentage increase equals 100 × instrumentation latency / application latency, e.g. a 2 ms span makes a 3 ms operation look ~67% slower but a 100 ms operation only ~2% slower. Measure in controlled production‑like environments with warmup and headroom, prefer sampling (some fleets use 1-in-1000), upgrade agents, disable expensive parsing (e.g. SQL attribute extraction) and avoid indiscriminate manual spans.
AI-written summary · about
chapters (7)
- 0:00Instrumentation overhead intro
- 1:48Defining overhead
- 5:20Latency changes misleading
- 9:54Overhead cost independence
- 12:00Measuring overhead in production
- 15:28Reducing overhead techniques
- 19:25Q&A
related talks
20mHow to Think About Instrumentation Overhead
27mInstrumenting at 10 years per second
51mProduction Profiling: What, Why and How
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
20mStrategies for Auto-Instrumenting Go Applications Without Code Changes
01h 00m“Source Instrumentation for Monitoring C++ in Production”
32mWhat's the Performance Overhead? Answering the Biggest Question in Tracing