Confidence in Chaos
summary
Practical checklist for chaos testing: measure the four golden signals—traffic, latency, errors, saturation—use real numbers (increase traffic from ~200 req/s to 500–600 req/s and watch latency jump from ~500 ms), define SLAs/SLOs and an error budget of 99.99% = 52 minutes/year, build correlated distributed logs and traces so failures are actionable, tune alert thresholds to avoid fatigue, start with auto-instrumentation, iterate instrumentation, and validate that alerts route to the owning team.
AI-written summary · about
chapters (9)
- 0:00Chaos engineering intro
- 1:04What is chaos engineering
- 3:11True observability attributes
- 5:40Golden signals of monitoring
- 7:05Actionable failure alerts
- 8:31Service level agreements and objectives
- 10:17SLAs in chaos testing
- 11:39Monitoring during chaos
- 13:46Observability closing thoughts
related talks
23mHow to tame chaos effectively?
30mReducing Chaos in Production: Testing vs. Monitoring
31mPlanning for Failure using Chaos Engineering
05mReducing Trauma in Production with SLOs and Chaos Engineering
48mChaos Engineering: The Path to Reliability
26mBreaking Systems for Resilience
45mDeveloping a Chaos Architecture Mindset
38mScientific and Safe Chaos Engineering