Ghosts in the Runtime: Who Ate My Capabilities & Other Mysteries
summary
Datadog runs tens of thousands of hosts across dozens of clusters up to 4,000 nodes and encounters low‑level Kubernetes mysteries with concrete fixes: a pod cache corruption comes from an in‑place slice sort mutating shared state and is fixed by copying before sorting (upstream PR), intermittent "/dev/urandom: operation not permitted" errors occur every ~10s due to runc’s cgroup device update race (resolved by upgrading runc), IPs stick because an ipvlan delete is skipped under CNI chaining while a long‑lived daemonset connection holds the netns alive (use short‑lived connections), and file capabilities break when CI image builds write v3 attributes with a userns root id (Docker 21 fix).
AI-written summary · about
chapters (5)
- 0:00Intro
- 1:09Disappearing Containers
- 9:40Random Failures
- 15:52Network Layer
- 24:04Who Ate My Capabilities
related talks
31mHow the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
27mWhat Doesn’t Kill You Makes You Stronger: The Vulnerabilities Th...
24mBuilding containers is fun. Let's see how?
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
32mUnderstand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar
38mLogs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS
34mReading 100+ Kubernetes Post-Mortems - lessons learned