Observability to Resolution: The Journey Through a Production K8s Incident
summary
A month-long production incident in a multi-cluster Kubernetes platform produces intermittent HTTP 500s; investigation shows ~70% of DNS queries are unresolvable due to ndots/search-domain expansion, flooding CoreDNS and breaking a proxy that relies on Kubernetes DNS. Using DNS-top, node-local cache (dnsmasq) logs and CoreDNS metrics reveals the pattern; fixes include raising node-local concurrent query limits, restoring the recursion header, and dropping known-unresolvable names, which cuts CoreDNS load from ~5,000 rps to ~1,000 rps and restores SLI targets. DNS-top and node-local (dnsmasq) tweaks
AI-written summary · about
chapters (12)
- 0:00Introduction and context
- 3:01The mysterious incident begins
- 5:11Initial misdiagnosis and fixes
- 8:43Recurring failures and frustration
- 14:46Building a useless dashboard
- 17:38Shifting focus to affected services
- 20:28Discovering the DNS latency issue
- 24:33First DNS fixes and partial relief
- 25:18Root cause analysis with DNS top
- 30:18Final DNS resolution and fixes
- 30:40Key lessons from the incident
- 35:44Q&A session
related talks
22mBest Practice: DNS Failure Observability and Diagnosis in Kubernetes
35mKubernetes, Lessons Learnt
35mGitOps or a Journey to a Production Incident and Swiftly Back Again
36mLogs Told Us It Was DNS, It Felt Like DNS, It Had To Be DNS, I...
32mEnhancing DNS Reliability in Kubernetes With Node Local DN...
29mKubernetes Failure Stories and How to Crash Your Clusters
38mCoreDNS: Intro and Deep Dive
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...