Visibility into Loggers (and Other Low Level Services)—Seeing the Trees from the Forest
summary
Measurements show file-based logging often costs 100–170 µs median with long tails out to 3.3 ms tail, while HTTP-backed logging moves medians into single-digit milliseconds and produced a max near 250 ms; strftime cost fell from ~1 ms a decade ago to ~15 µs today but still exceeds a syscall. A malloc bug grew a free list to ~1M entries, degrading CPU over a week. Proposals: expose in-process metrics via a /proc-like interface, or use LD_PRELOAD interposing to instrument low-level libraries.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:34Credentials
- 1:44Outline
- 2:19History of Fire Codes
- 4:48Logger Moment
- 7:22Measuring Latencies
- 8:21Sample Data
- 8:48Log Back Logging Example
- 12:47Malloc Moment
- 15:32Low Level Services
- 16:21Metrics vs Logging
- 18:53Processing Application Metrics
- 22:09Visibility into Low Level Services
- 27:16Questions
related talks
35mHigh Throughput with Low Resource Usage: A Logging Journey
31mMy Logs Cost More Than My Services
38mFighting latency: the CPU profiler is not your ally
42mHow Much Is Tracing? Measuring the Overhead Caused by the Tracing Infrastructure
56mContinuous Profiling the missing piece in your observability puzzle
44mObservability Tools C++: Beyond GDB and printf - Tools to Understand the Behavior of Your Program
39mFast log management for your infrastructure
38mLogging in the age of Microservices and the Cloud