Gain a deeper understanding of your incidents: continuous profiling
summary
Continuous profiling with Pyroscope and pprof pinpoints real root causes: an incident shows 800k goroutines chewing gigabytes of memory due to per-line logging locks, fixed by removing those locks and adding log rate limits; another shows heavy CPU in regex evaluation on the ingester, fixed by limiting regex length and query parallelism and planning read/write separation. The demo uses Grafana Agent and fgprof/pprof, reports typical overhead of 2–5% overhead in busy clusters and higher on low-traffic services.
AI-written summary · about
chapters (10)
- 0:00Introduction to continuous profiling
- 1:05Observability signals overview
- 4:17Why continuous profiling matters
- 5:40Use cases for profiling
- 12:27Profile types explained
- 17:01Real incident investigations
- 23:25Demo setup and CPU profiling
- 28:23Flame graph diff analysis
- 31:41Q&A on diff view and overhead
- 37:47Q&A on continuous vs static profiling
related talks
24mCommunity Driven Breakout Session: Continuous Profiling
56mContinuous Profiling the missing piece in your observability puzzle
01h 13mContinuous Profiling: The Missing Piece in Your Observability Puzzle
36mContinuous Profiling in the Cloud-Native era
26mProfiling Mastery: Decode Your Data and Elevate Observability
40mContinuous Profiling, the missing piece in your observability puzzle!
28mConprof - Profiling in the cloud-native era
36mProfiling in the cloud-native era - Matthias Loibl