Spotify’s outage of 8.3.2022, explained
summary
An evening outage on 8 March 2022 lasts about an hour and is traced to service discovery: Spotify runs a mix of Traffic Director (GCP) and a DNS-based nameless system, and an error in the gRPC Java resolver causes upstream name-resolution failures to be propagated as NOT_FOUND. Login services interpret NOT_FOUND as missing users and revoke credentials, disrupting roughly 50 million sessions and later creating ~3 million duplicate accounts. Mitigation uses reverts to nameless and a gRPC client fix that maps errors to UNAVAILABLE.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:45Kats background
- 2:52The outage
- 4:35Monitoring
- 5:23User impact
- 5:46Full scale outages
- 6:11Spotifys tech stack
- 8:02Spotifys architecture
- 8:57Service discovery
- 11:05Traffic Director
- 12:34Logs
- 13:33The solution
- 14:16Why were users logged out
- 15:04Why was login service not found
- 15:46The fix
- 16:49The aftermath
- 17:37Lessons learned
- 19:35Login vs email
- 20:30Fun
- 20:50Apple Event
- 21:10Radio
- 21:29Kirka
- 22:13Conclusion
related talks
34mThe Spotify Story: Moving from NIH to an Active CNCF Community Member
20mSpotify's Growth Exposed the Abstraction Paradox
09mSelf-Service Infrastructure
40mThe World Blew Up but We’re All Okay: How We Managed a...
14mHow We Implemented High Throughput Logging at Spotify
24mLessons Learned from the Halloween Outage
26mThe Sounds of Silence: Lessons from an 18 hour API outage
26mThe Big One