Logs Told Us It Was DNS, It Felt Like DNS, It Had To Be DNS, I...
summary
Rolling restarts spike client retries and latency; logs point to DNS but low-level ENA metrics and VPC flow logs show conntrack table exceeded and hundreds of thousands of SYNs. Debugging finds SYNs dropped by Linux reverse path filtering because pod secondary interfaces lack IPs, and DNS/route propagation is quantized by external-dns TTL (15s) plus gRPC resolution timing (30s). The SYN flood stems from thousands of clients using a 300ms reconnect backoff; the fix is to stop that aggressive backoff — deleted three lines — and restore sane behavior.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:47Metries service errors during rollouts
- 2:52Applications involved
- 3:57DNS setup
- 6:15Too many queries at startup?
- 8:11Networking issues?
- 12:01Let's test with network optimized instances
- 12:33What about bigger instances?
- 14:04VPC Flow Logs
- 15:58Zoom on ingress flows to old IP
- 16:43What about egress?
- 18:42Routing on nodes
- 19:43Stable state
- 20:09What about traffic to old IP?
- 20:36Let's simulate
- 21:42Reverse Path filtering
- 25:532 questions
- 26:06RPC setup
- 26:38DNS propagation time during Rollouts
- 32:03Reconnection differences
- 34:37Lessons Learned
related talks
39mObservability to Resolution: The Journey Through a Production K8s Incident
22mWhy is it always DNS, TLS, and bad configs?
31mWhy is it always DNS, TLS, and bad configs?
35mKubernetes, Lessons Learnt
30mNoisy Neighbors, through Networking
22mBest Practice: DNS Failure Observability and Diagnosis in Kubernetes
45mHello from the other side
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap