Testing in Production
summary
Company moves to commit-to-master with no branches, synchronous five-minute deploys driven by a tiny bash script, and one-click rollback by checking out HEAD-1. Slow functional and end-to-end tests run post-deploy functional tests on production via parallel cron groups that alert on failures; unit tests gate deploys. Feature flags and physical deploy tokens serialize deploys. Load tests amplify real user events (up to ~40x), auto-stop on wobble after a previous incident, and metrics flow to Graphite/Grafana with nudges for alerts.
AI-written summary · about
chapters (35)
- 0:00Intro
- 1:41Conventional deploy systems
- 3:44Hello. My name is...
- 4:04Agile and XP
- 4:39Ad tech 101
- 6:24Our goals
- 8:02Ingredients to increase release cadence
- 9:17Faster deploys
- 10:04Commit to your master
- 10:47Commit == Deploy
- 13:50Welcome to the menagerie
- 15:33Taking the shortest route
- 16:10Synchronous or async deploy?
- 19:53Feedback loops
- 22:19Flavours of testing
- 25:33Timing is everything
- 28:22Testing against production!
- 29:07Running functional tests... against production
- 30:54Special casing
- 32:30Users in the wild
- 33:50Load testing... in production
- 35:26Load patterns
- 37:24Sharks and waves
- 39:40Rolling back
- 42:33Overlaying events
- 44:19Too many metrics!
- 46:39Monitoring Driven Development
- 47:35Alerting
- 47:50What to alert on?
- 49:46Feedback loop for testing in prod
- 50:13"Full cycle" developers
- 51:37Building great teams
- 52:30Developer happiness
- 52:53Do try this at home...
- 54:47Thank you!
