Making Swiss Cheese of Ticket Sales

on YouTube ↗30m

" summary

Custom ticketing system built on TypeScript React Node Express MySQL on Kubernetes collapses two minutes into a launch, selling one ticket in the first 100 minutes. Root causes are concrete: exhausted DB connection pools, payment-record creation inside a transaction that disappears on container restart, CPU-blocking PDF generation, and nginx 60s timeouts that let customers get charged up to 12 times. Fixes include raising MySQL limits and CPU, consolidating transactions, disabling row locks, offloading PDF work, and heavier load testing — next launch succeeds.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 0:17Making Swiss Cheese of Ticket Sales
3. 5:43Architecture for the Ticketing System
4. 7:11The Swiss Cheese Model
5. 16:31Diagnosing Optimizing Sql
6. 22:06Resource Limits
7. 23:51Why Did We Have So Many Users Trying To Buy Tickets
8. 26:58Human Factors
9. 29:40Planning for Failure Using Chaos Engineering

" related talks

• 31mDOES14 - Shakeel Sorathia - TicketmasterShakeel Sorathia · DevOps Enterprise Summit 2014 (San Francisco 2014)
• 40mZero-downtime payment platformsPrem Sichanugrist, Ryan Twomey · RailsConf 2013
• 35mDo the holes in Swiss cheese leak abstractions?Tim Bell · DjangoCon Europe 2023
• 08mMonitoring Errors and Slowdowns with a JS Frontend and Node BackendChris Stavitsky · Node Congress 2022
• 38mBuilding a high performing Passenger Information SystemRobbert Noordzij · J-Fall Virtual 2020
• 29mSupercharging Node with KrakenTim Messerschmidt · JSConf EU 2014
• 28m193 Easy Steps to DevOpsing Your MonolithCat Swetel · DevOpsDays Seattle 2019
• 34mNightmare Before Best PracticesJose Castro · Scala Days Berlin 2016