Case Study: A Thundering Herd in the Wild
summary
Investigation traces extreme P95 outliers to thousands of unrelated threads waking simultaneously every 5 seconds due to recurring jobs aligned to the clock, not an overloaded datastore. Workflow uses an on/off‑CPU sampler, dynamic instrumentation, a granular-sample patch, a simple Python lag detector and an eBPF core sampler to prove gaps aligned to the system clock. Root cause is libraries using time job scheduler callbacks with absolute timers creating a thundering herd. Fixes (relative timers, jitter, prime delays, subscribe-only wakeups) remove the spikes.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:59Defining thundering herd
- 4:15A performance mystery
- 7:21Profiling with sampling
- 12:32Using code instrumentation
- 19:42Granular sampling insights
- 24:54Root cause identification
- 27:35Solutions and conclusions
related talks
19mDo nothing fast: How to scale idle cpus?
30mAbove the Clouds: Investigating Systems from 32,000 ft.
49mNoisy Neighbor Detection: A New OSS Collector
37mEverything you ever wanted to know about threads...
20mUnderstanding Request Latency with Wallclock Profiling
30mNoisy Neighbors, through Networking
18mScheduler Tracing With ftrace + eBPF
54mUnderstanding Performance w/DTrace (While The Customer Yells)...