Avoiding Cachepocalypse in the Land of the Monolith
summary
Legacy Python monolith exhausts memcached capacity because one cache node receives duplicate connections, pushing a normal 32,000 connections per node to over 64,000 and blocking horizontal scaling. Debugging uses local tools including a Docker lab and native debugger, uncovers a memcache client mismatch and metadata flag incompatibility between versions that corrupts Spanish data, and shows that setting options after initialization leaves lingering memcached connections. Fix is to pass options in the constructor, roll out gradually, and recover after a 45-minute cache flush.
AI-written summary · about
chapters (10)
- 0:00Legacy monolith challenges
- 0:49Monolith architecture overview
- 1:51Cache connection bug
- 3:15Stakes and team alignment
- 4:17Debugging staging
- 6:24Spanish launch in production
- 7:48Outage root cause
- 9:15Fixing cache issue
- 10:16Incident lessons
- 11:21Long-term improvements
related talks
27mWhen Linux Memory Accounting Goes Wrong
34mThe Missing Metrics: Measuring Memory Interference in Cloud Native Systems
29mTrip Down Memory Lane: OOM in Python
39m30 minutes of Failure
22mPinpoint Ceph Bottleneck Out of Cluster Behavior Mists
16mMicroMon: A Monitoring Framework for Tackling Distributed Heterogeneity
28mCommon Support Issues And How To Troubleshoot Them
42mPlanning To Fail