Practicalities of Productionizing Distributed Systems
summary
Covers concrete tactics: use percentiles not averages for latency, add distributed tracing to follow one request across services, and expect many partial failures like GC pauses and sticky overloaded processes. Recommends feature flags (hash+modulus) called The Cider to decouple deploys from releases, uses overload responses and backpressure (drop or reject with metrics), and describes anti-spam heuristics, regex rules, IP hashing and schedulers (Kubernetes) with real production lessons and trade-offs.
AI-written summary · about
chapters (30)
- 0:00Intro
- 1:09Why you should listen to me
- 2:36Quick foundation
- 3:03A subset of failures
- 3:21Clients stuck to an overloaded process
- 3:53Partial failure
- 4:12"It's slow" is the hardest problem you'll ever debug
- 4:46Create partial availability
- 6:45"Who to Follow" in the monorail
- 8:54Knowing what the system has done
- 9:59Percentiles, not averages
- 11:02Tracing
- 12:20On profiling
- 14:26Releases should change a metric
- 15:11Free-form logs are liars
- 15:35Common "problems" are overlogged
- 16:26Uncommon problems
- 17:05Avoid coordination
- 21:32Backpressure
- 23:28Dropping new messages on the floor
- 23:55Returning "overload" error responses
- 24:50Timeouts and exponential back-offs
- 25:35Roll out infrastructure with feature flags
- 26:21if (Decider.available..)
- 29:36Multiple versions are the norm
- 30:24Datacenter schedulers are worth it
- 32:04Collaboration is politics
- 33:08No time-traveling stalkers
- 34:41moral necessity
- 34:58Data minimization is a
related talks
43mGetting Your System into Production and keeping It There
29mPractical distributed systems: Building for the real world with chaos engineering
03mCultivating production excellence
58mCultivating Production Excellence
25mAvoiding Overload in Distributed Systems
31mGo From Dev to Prod
19mThink Prod Up Front: Shifting Left on Production Excellence
46mDeploying and maintaining software for today's infrastructure