The Definition Of Insanity
summary
Argues that timeouts, retries, idempotency are the real levers for resilient distributed systems and gives concrete fixes: tune histograms-first timeouts (avoid default 30s and connection-pool wait -1 that blew a site from 30–60 to ~800 threads under 8,000 rps), cap retries and add jitter (Square’s incident: 500 hard‑coded retries hammered Redis), use libraries like Polly or Resilience4j, prefer client request IDs or server-side MD5 fingerprinting with expiry to make retries safe and avoid doubling payments.
AI-written summary · about
chapters (23)
- 0:00Welcome
- 1:12Resilient distributed systems
- 2:35Distributed systems defined
- 5:43Two golden constraints
- 6:03Timeouts fundamentals
- 8:31Timeout misconfiguration
- 13:49Balancing timeouts
- 14:30Histograms for response times
- 17:18Consistent latency matters
- 18:21Tail latencies dangers
- 22:12Why retries make sense
- 24:39Implementing retries
- 26:02Square’s latency incident
- 27:24Avoiding retry waves
- 30:13Idempotency challenge
- 33:21Item potency explained
- 34:25Request IDs for idempotency
- 36:30Server-side fingerprinting
- 43:49Safe retries with responses
- 45:14Timeouts, retries, idempotency
- 46:38Einstein insanity quote
- 47:20Final recommendations
- 49:49Q&A
related talks
44mArchitectural Patterns of Resilient Distributed Systems
01h 00mSystem Stable: Robust connected applications with Polly, the .NET Resilience Framework
49mManaging Resiliency in Microservices
43mDistributed Systems
58mResilient by Design
29mFault Tolerance is a Requirement Not a Feature - Adar Danait, Lilian Ernest, Citi
40mLet it fail: resilience patterns for microservices
48mFault Tolerance on the Cheap Making Systems That Probably Won't Fall Over