Trustworthy Graceful Degradation: Fault Tolerance across Service Boundaries
summary
Third of Australian students use the service, with tens of thousands online simultaneously, and a default 30 second timeout once makes the app feel totally down. Deploys run Selenium smoke tests and blue‑green rollouts; unit tests in CI spin up a real MySQL to force “too many connections” and validate reconnection logic, and the Redis client includes stampeding‑herd protection. A header-based request cursing system (e.g. X‑Style‑Curse‑Mongo‑Requests) simulates downstream failures per request so CI and production can verify graceful degradation.
AI-written summary · about
chapters (13)
- 0:00Intro
- 1:22A Highly Available Web Service Simple, elegant, effective
- 3:20Postmortems and Action Items to the rescue! Establish the root cause.
- 6:29Failing gracefully: Understand all of the possible error modes
- 8:01Failing gracefully: ...and fail helpfully where you can't
- 10:26Failing gracefully: First do no harm Don't make it worse by retrying reconnecting and restarting
- 12:50When fixing the problem isn't enough Many of you are thinking: Chaos engineering can help here!
- 13:27Unit test your service clients
- 15:01Integration test your minimum viable system
- 16:44Chaos Testing If you think you don't need a service, try shutting it down. In prod. Right now
- 17:063. Chaos Testing: Limitations
- 17:45Advanced Technique: Request 'Cursing' How to get the benefits of chaos testing optional dependencies without actually degrading the experience for your users.
- 18:244. Request Cursing: The Result
related talks
43mFault tolerance – look, it’s possible!
32mHow does your application behaves when everything goes wrong?
53mResiliency and Availability Design Patterns for the Cloud
15mBoosting System Resilience: Exponential Backoff
45m⚡A Resilient Application in a Partially Degraded World ⚡
34mDistributed Systems - What Can Go Wrong and How to Get It Right
51mBuilding Fault Tolerant Microservices
28mBuilding and testing resilient applications