Black Friday - Lessons in Resiliency and Incident Response at Shopify
summary
Shopify describes a Black Friday cascade: two rack PDUs fail, engineers fail over two pods to a shadow data center and hit $1M per minute in sales within minutes. Root causes: Redis out‑of‑memory plus host‑level monitoring and a read‑based health check that hides write failures, which triggers an N² worker recovery and a 15‑minute delay. Changes: instance‑level metrics, write health checks, Semian circuit‑breaker tweaks and automated resiliency tests using Toxiproxy and enforcing three copies of persistent data.
AI-written summary · about
chapters (7)
- 0:00Introduction to Black Friday
- 1:55Resiliency over avoiding failures
- 4:44Principle one: Redundancy everywhere
- 9:28Principle two: Multiple levels
- 16:10Degraded is better than down
- 18:23Post-mortem and lessons learned
- 24:22Q&A session
related talks
32mSurviving Black Friday - A resilience engineering tale
42mHow to stay alive even when others go down:...
06mIgnite: How DevOps Antipatterns Ruined Holidays
33mShopify in Multiple Datacenters
35mHow to Survive Black Friday: Preparing e‑commerce for Peak Season
33mResilient
40mHow Shopify Scales Rails
40mWhispers in Chaos: Searching for Weak Signals in Incidents