Finding the Capacity to Grieve Once More
summary
Wikipedia suffers a major outage in 2020 when a celebrity death converges with a DDoS, exposing parse-thundering and cache invalidation bugs; fixes include migrating MediaWiki to Kubernetes, deploying DDoS scrubbing suppliers, adding a distributed in‑memory parse lock, adopting Envoy service mesh, running ma-router from Meta for memcache routing, switching to Cassandra for sessions, moving cache invalidation to Kafka and enabling multi‑DC active use, and the result handles spikes (sampled 6.5 million peak) without collapse.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:21Wikipedia’s infrastructure
- 4:27MediaWiki’s scale
- 6:38Michael Jackson effect
- 9:48Cache stampede fixes
- 11:57Celebrity deaths impact
- 14:28Anxiety from crises
- 16:11Kobe Bryant incident causes
- 17:37Post-incident response
- 19:25DDoS protection role
- 21:10Wikidata regression fixes
- 23:17Multi-data center changes
- 24:44Session store optimizations
- 28:36Incident response culture
- 30:22Testing with Elizabeth II
- 32:53Closing remarks
related talks
16mThe Wikimedia infrastructure- 16mThe Wikimedia infrastructure
31mSeeing the Invisible: Two Years at Wikipedia with W3C's Network Error Logging
26mA Tale of Two Outages
25m"The website's down!" Stories and lessons on keeping your website up.
10mGreat Caching Disasters!
21mHandling a Major Outage Transparently
34mDatacenter Fires and Other 'Minor' Disasters