Who Killed My Pod? #Whodunit
Talkat KubeCon + CloudNativeCon North America 2021 · on Wednesday, 2021-10-13· at 15:25· Petree Hall C + Online·
summary
A newly deployed, well‑tested app on a self‑managed Kubernetes cluster repeatedly gets OOMKilled with exit code 137, crashing into CrashLoopBackOff. Investigation traces kills to the Linux OOM killer (not kubelet), cgroups reporting ~31.38 GB anonymous/hugepage usage against a 31.45 GB container limit, with swap disabled. Setting vm.overcommit_memory=2 on affected nodes mitigates the failures; the long‑term fix is reducing the app’s memory footprint and matching pod requests/limits so guaranteed QoS prevents future unpredictable OOM kills.
AI-written summary · about
chapters (9)
- 0:00Start
- 1:06The serial killer investigation board
- 1:35Investigation: Container runtime (CR)
- 2:58Investigation: Kubernetes & CRI
- 5:17Investigation Why
- 16:33Kube: QoS & Resource requests & limits
- 18:01Kube: Resources & commitments
- 19:00Over-committal: It's a feature not a bug!
- 20:08Data from crime scene!
related talks
45mCatching the 137-Killer: A Java Memory Forensics Investigation
29mKubernetes Failure Stories and How to Crash Your Clusters
34mReading 100+ Kubernetes Post-Mortems - lessons learned
30mWhat we learned from reading 100+ Kubernetes Post-Mortems
27mWhen Linux Memory Accounting Goes Wrong
39mStaying in Tune: Optimize Kubernetes for Stability and Utilization
34mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production
36m101 Ways to Crash Your Cluster