How Netflix does failovers in 7 minutes flat

on YouTube ↗32m

" summary

Netflix claims it can shift all 100M+ users under seven minutes by prebuilding nimble shadow clusters that Python orchestration (Flask, Redis, async queues) keeps topped up using Atlas and Edda caches, Cassandra for replication, and an edge proxy plus DNS cutover. Detection drops to ~2 minutes, scaling becomes ~4 minutes by transplanting ready instances, DNS ~3 minutes, down from ~45. The system runs in monthly “chaos Kong” exercises, is cost‑neutral using reserved instances, and reliably works in production.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 5:08Prerequisites
3. 6:32Failover Candidate
4. 8:14Regional Failover Process
5. 11:24Scaling Pattern
6. 12:39Traffic Shift
7. 15:06us-west
8. 19:10Periodic Tasks
9. 19:44Triggered Tasks
10. 20:05Nanoservices
11. 21:00Characteristics
12. 22:23Anticipate Failure
13. 22:56Fallbacks
14. 23:42Eventual Consistency
15. 24:18Orphan Cleaner
16. 27:18Suggestions

" related talks

• 31mKeeping Your Cloud Footprint in CheckCoburn Watson · GOTO London 2015
• 50mFast DeliveryAdrian Cockcroft · GOTO Copenhagen 2014
• 31mHow Netflix Thinks About DevOpsDave Hahn · DevOpsDays Rockies 2016
• 27mBehind the Scenes: Running Critical Services @ NetflixManuel Correa · ScaleConf Colombia 2019
• 29mFast, Safe, and Continuous -- Picking all 3 at NetflixDave Hahn · Spinnaker Summit 2019
• 41mHow Netflix Embraces Failure to Improve Resilience and Maximize AvailabilityAriel Tseitlin · LISA '13
• 55mIncident Management at Netflix VelocityDave Hahn · LISA18
• 29mHow Netflix Thinks About DevOpsTejas Chopra · Developer Growth Summit 2022