Slack's DNSSEC Rollout: Third Time's the Outage
summary
A large SaaS enables DNSSEC across dozens of domains, then trips on three failed attempts to turn it on for slack.com: first aborted for an ISP outage, second rolled back when VPN-linked resolvers reject apex CNAMEs, third breaks more users after pulling the DS because the .com zone caches that DS for 24-hour DS TTL caching so rollback doesn’t stop validation. Root cause is a Route 53 wildcard NSEC bug that returns incomplete NSEC bitmaps, causing NO DATA responses to be cached by aggressive resolvers; remediation: bug fix, runbooks, extra logging, TTL reductions, and a cautious fourth rollout.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:43DNSSEC Basics
- 4:23Slacks Journey
- 6:09First Attempt
- 17:29The Mystery
- 18:00What Happened
- 19:01Analysis
- 21:57Planning for Attempt 4
- 25:58Thank you
