Fighting latency: the CPU profiler is not your ally
summary
CPU sampling misses wall‑clock latency caused by blocking so a service that looks CPU‑light still shows high request times; using runtime/trace + net/http/pprof records nanosecond events (network, file close, goroutine scheduling) and shows that repeated file.Close calls account for seconds of blocked time while the CPU profile blamed hashing. Traces are large (tens of GB) and add ~25% overhead, so author exports pprof from traces and ships a trace-focus tool to attribute blockers across spawned goroutines, and it works.
AI-written summary · about
chapters (9)
- 0:00Defining performance
- 3:52Understanding CPU profiling
- 6:13Scheduler and I/O visibility
- 11:35The limits of CPU profiling
- 15:16Introduction to the tracer
- 21:27Exporting trace data
- 24:09Analyzing blocking profiles
- 27:13Challenges in trace analysis
- 31:50Building custom trace tools
related talks
44mGo Profiling and Observability from Scratch
25mProfiling Request Latency with Critical Path Analysis
20mUnderstanding Request Latency with Wallclock Profiling
51mProduction Profiling: What, Why and How
28mDeep Dive Into Profilers: How CPU Profilers Measure Your Application's Performance
29mProfiling Node Applications
26mProfiling Mastery: Decode Your Data and Elevate Observability
25mPerfect Match: Correlating Continuous Profiling with Distributed Tracing for Stronger Observability