Reducing Chaos in Production: Testing vs. Monitoring
summary
Claim: testing can’t catch scale and user-driven failure modes, so pair targeted tests with instrumentation that surfaces business impact. Case study: a client with 100 million users and one billion emails a month (≈300k lines of code) collects ~5600 metrics and traces revenue, traffic, latency and email bounces; a revenue dip tracks to rising email bounce rates, not servers or DB. Recommendation: monitor business-facing metrics, correlate them with system metrics, and only alert on actionable failures.
AI-written summary · about
chapters (18)
- 0:00Start
- 0:58a discussion on trade-offs between testing and monitoring and how to apply them in a devops lifecycle
- 3:19testing is required
- 4:03unit testing functional testing resilience testing performance testing
- 4:38testing can give a false sense of security
- 5:20testing is deterministic
- 5:46data problem
- 8:37user problem
- 9:06the Corrupted Blood incident
- 11:48lack of foresight too many use-cases change to assumptions
- 13:26100% code coverage is not the goal
- 14:31testing is good for "known knowns"
- 15:15testing is "unknown unknowns"
- 15:37why monitor?
- 28:34testing and monitoring not
- 28:46understand the business
- 29:00increase observability (metrics, tracing, etc..)
- 29:21alert only on actionable emergencies
related talks
28mProduction Testing through Monitoring
28mProduction Testing Through Monitoring
15mConfidence in Chaos
58mObserving Applications in Production
19mDo more with less: Unit and integration testing is broken
15mWhy I Prefer Good Testing Over Excellent Testing
56mTesting at scale
58mCultivating Production Excellence