Lessons Learned The Hard Way
summary
Migration from integer to float(3,2) truncates values so 10 becomes 9.99 and thousands of rows corrupt, recovery from backup takes hours; resetting a test sequence leaves customer IDs >1,000,000 and a typo in an update script rewrites all rows; a 512 MB queue worker crashes when millions of images backlog the queue; exchange-rate API key invalidation silently yields zero rates for weeks; Django autogenerated migration drops/recreates foreign keys and blocks large tables. Key actions: backups, monitoring, timeouts on network calls, staged deploys, and check migrations SQL while remembering replica is not a backup.
AI-written summary · about
chapters (21)
- 0:00Introduction and speaker intro
- 0:48First production crash mistake
- 3:41Never change between test and prod
- 4:02Invalid customer IDs in accounting
- 5:47Never run untested code on production
- 6:29Replica is not a backup
- 7:11Importance of monitoring and alerts
- 8:35Running out of 32-bit IDs
- 10:42Always communicate major changes
- 12:51Queue overload from image processing
- 15:42Third-party API reliability issues
- 18:49Always lock third-party errors
- 19:11Monolith to microservices migration
- 22:50Read error logs carefully under pressure
- 24:15Always set network call timeouts
- 26:00Time your deployments wisely
- 27:46Avoid last-minute production deploys
- 28:07Django migration foreign key issue
- 31:39Always check migration SQL
- 32:19Final advice: Don't panic
- 33:42Q&A session begins
related talks
43mFrom Developer to Development Manager
30mautopsy-of-a-slow-train-wreck-the-life-and-death-of-a-django-startup/
22mLearnt lessons in a big Django Project
19mLearning from eight years of data science mistakes
26mThe Butterfly Effect: How a single bit changed my career
48mCtrl-Alt-Del: Learning to Love Legacy Code.
40mMistakes were made: Data science ten years in
39mConfessions of a Systems Engineer: Learning from My 20+ Years of Failure