What can possibly go wrong? The story of an outrage at Booking.com
summary
A production cron job running every six hours has a bug that gradually removes availability and silently leaks bookings; a code change lands while the previous long-running job is still executing so rollback plus a restore from backup and data-repair are required before normal traffic returns. Monitoring shows platform-level divergence not service errors, emergency workflow activates a panic button, around 50 engineers join under an incident leader, and recovery completes that night.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:01Easter Friday outage begins
- 3:51Booking.com and scale
- 7:34Technology stack complexity
- 12:36Monitoring and incident response
- 14:02Panic button and no-blame culture
- 17:17Diagnosing leaking bookings
- 19:05Root cause: faulty Cron job
- 20:14Restoring data and recovery
- 21:02Lessons learned
related talks
48mWhen things go wrong
34mDatacenter Fires and Other 'Minor' Disasters
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
21mHandling a Major Outage Transparently
40mThe World Blew Up but We’re All Okay: How We Managed a...
30mIncident Groundhog Day
57mThe Curious Incident in your Software in the Day Time
45mWhat to do when complex systems fail: A guide to chaos management