Post Mortems for 4 Years of Remote Execution
summary
Four concrete incidents from four years of running remote execution: a deadlock caused by single outgoing HTTP/2 connection per worker and cyclic RPCs that exhaust streams, an API server crash from missing flow control while proxying large file downloads, intermittent slow builds caused by an aggressive autoscaler plus a logging-induced CPU spike in a GMA proxy, and a pool outage where 172,000 compile actions causing queue overload exposes the need for timeouts, quotas and cancel APIs.
AI-written summary · about
chapters (8)
- 0:00Remote execution post mortems
- 2:06What a post mortem entails
- 3:51Cyclic RPC deadlocks in workers
- 6:21API server crashes from flow control
- 9:11Slow builds from GMA server issues
- 16:19High queue age from client overload
- 20:36Incident root causes and lessons
- 23:49Q&A on best practices
related talks
50mBuilding on Key – Keeping your Actions and Remote Executions in Tune + Q&A
39m30 minutes of Failure
11mRemote execution with 100 to 100,000 CPUs
22mAfter Death
33mWhat We Learned from Reading 100+ Kubernetes Post-Mortems
33mA Tale of Two System Crashes: Innovation and Accidental Chaos
51mHearts of Darkness: A Spring DevOps Apocalypse
42mArchitecting for Failures in Microservices