Logs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS
summary
Kubernetes pods hit intermittent DNS failures during rolling restarts while node-local-dns OOMs despite increasing memory from 64→256 MiB and max_concurrent from 1k→2k. ENA metrics and VPC flow logs reveal conntrack table saturates (stable ~13k connections, spiking to ~60k) caused by tens of thousands of SYN retries from alerting clients. Root cause is client behavior: gRPC round_robin + aggressive reconnect backoff (300 ms) reopens connections during a 10 s graceful shutdown and 30 s min-resolve window. Fix: remove the aggressive reconnect overrides and add a small CNI route change.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:58Kubernetes
- 1:42How It Started
- 3:13What You Always Do
- 3:28DNS
- 10:16Summary
- 10:33AWS Networking
- 18:11Node Networking
- 22:59Reverse Path Filtering
- 27:10RPC Setup
- 31:47GRPC History
- 34:30GRPC Load Balancing
related talks
35mKubernetes, Lessons Learnt
30mThe Unconspicuous Role of Conntrack in Kubernetes Networking
28mHow to Break your Kubernetes Cluster with Networking
06mLightning Talk: Is Your Kubernetes Cluster's DNS Working?
32mUnderstand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
31mHow the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales
27mCoreDNS Deep Dive