The Danger of Implicit Blocking in Finagle
summary
Finagle services block when long computations or explicit waits run on Netty event-loop threads, causing worker threads (default 2× logical CPUs) to exhaust and latency to jump from ~200 ms to ~2 s (up to 10×) under load. Hidden symptoms include nonzero blocking_ms and rising handle_time_microseconds. Fixes: remove Await.result/Await.ready, move CPU work onto a separate FuturePool sized for the machine, beware Twitter vs Scala future semantics, and avoid unbounded thread pools that can OOM.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker
- 1:01Implicit blocking explained
- 2:49Production latency spike
- 3:32Explicit blocking demo
- 4:16Implicit blocking demo
- 5:43Finagle thread model
- 8:07Worker thread exhaustion
- 9:12Key metrics to monitor
- 10:16Solving the problem
- 12:36Conclusion and takeaways
- 13:29Q&A session
related talks
58mTurbo Charge CPU Utilization in Fork/Join Using the ManagedBlocker
53mTurbo Charge CPU Utilization in Fork/Join Using the ManagedBlocker
37mConcurrency options on the JVM
43mA Tale of Two Cities: How Blocking Calls Are Treated?
35mUntangled: Sneak peek at Project Loom
50mThinkAsync in Java8
52mMicro-Optimization for Fun and Profit
47mVirtual Threads power with Helidon Nima