Bug of the Year: A Memory Leak Reboot Could Not Help with
summary
A Ceph OSD memory leak stems from corrupt PG dupes entries that persist on disk so reboot does not help and an OSD must be redeployed to recover; the root cause is a trimming bug triggered by PG splits plus a libstdc++ std::list::size complexity mismatch and was amplified when autoscaler defaulting changed in Octopus. Operators see millions of dupes entries, TCMalloc warnings and latency spikes. Fixes include an injector to reproduce the state, new admin checks (mempools, PG query), and an offline tool trim-pg-lock-dupes plus phased OSD-side trimming patches.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker background
- 1:08Memory leak immune to reboot
- 9:53Symptoms and diagnosis heuristics
- 12:51What is PG log and dupes issue
- 20:09How dupes entries are generated
- 25:01Auto scalar and split operations
- 28:17Root cause and recent triggers
- 30:06Workarounds and prevention
- 33:45Fixing the problem challenges
- 38:23Offline and online streaming fixes
- 41:40Conclusion and Q&A
related talks
19mIdentifying and Eliminating Contention from Booting Concurrent SNP VMs
27mWhen Linux Memory Accounting Goes Wrong
30mdrm and cgroup memory management - two great tastes
45mIn-depth technical story: Fixing I/O performance for Windows guests in OpenStack Ceph clouds
35mAll Your Queues Are Belong to Us: The Hunt for a Network Bug...
22mEMD: Fair and Efficient Dynamic Memory De-bloating of Transparent Huge Pages
16mTaming Zombie Cgroups: Should Reparenting Be Simple or Smart?
01h 36mTutorial: Linux Memory Management and Containers