Pixels, Promises, and Panic - Horror Stories of Production Nightmares
summary
Real-life failures and fixes: a browser game bug (caused by one line of PHP) inflates scores into the billions and bans 20 children before accounts are restored; a misassigned BigQuery billing account generates a $50,000 bill that gets fixed after escalation; an auth redirect loop creates 3 million analytics events and a $25k spike before vendor forgiveness; a CSS loader left on an Angular checkout quietly destroys conversions. Concrete remedies: thresholds and alerts, per-user event limits or proxying, visual CSS tests, rollback + postmortems.
AI-written summary · about
chapters (9)
- 0:00Introduction to production nightmares
- 3:19Milk campaign and child leaderboard chaos
- 11:5150k bill from Google Cloud scare
- 16:25Infinite redirect loop and analytics bill
- 18:10CSS button breaking checkout revenue
- 21:23Glovo checkout CSS removal disaster
- 24:56Refactoring horror stories
- 25:58App crash under DDoS-like attack
- 30:41Lessons learned and closing remarks
related talks
41m12 Factors of Pain and Suffering
44mProject Triage: What to Do When Everything Hits the Fan
39mBehind the Game: Strategies for Building the Tech That Powers Today’s Top Titles.
46mBlink and You’ll Miss It: Stories From Discovering Most Annoying Bugs and Test Flakes
22mFear of the computer
35mDebugging the Mind: A Programmer's Guide to Critical Thinking
52mEscaping Developer Nightmares
30mReporting Bugs: Errors Made and Lessons Learned