How We Found a 2 GB/Day Leak in 5 Minutes – dotMemory Meets Akka.NET
summary
A real production leak reproduces locally and gets diagnosed with dotMemory and Rider in minutes: a throttler using ask creates temporary future actor ref instances that never emit deathwatch notifications, so a watched-entry dictionary retains task and TaskCompletionSource objects and accumulates roughly 2 GB/day in affected deployments. A small code fix to send Terminated for future refs ships in the SQL plugin 1.3.8 and stops the leak.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 0:28Akka.NET and actor model
- 1:11Persistence and throttling
- 3:18Memory leak in SQL
- 5:49User evidence and graphs
- 7:35Short-lived objects as signals
- 9:19Reproducing in test
- 11:27Setting up dotMemory
- 13:35Observing memory growth
- 15:02Analyzing snapshots
- 17:05Death watch retention issue
- 19:12Decade-old bug found
- 20:16Fix for death watch
- 23:52dotMemory uncovered bug
- 26:24Fixing Akka.Persistence.Sql leak
- 27:27Memory leak troubleshooting
- 31:21Profiling in production
- 34:32Actor lifecycle management
- 36:18Benchmarking memory issues
- 38:05Proactive vs reactive profiling
- 41:19OpenTelemetry diagnostics
- 43:47AI tools for leaks
- 47:20Q&A
