Have You Tried Turning It Off and Turning It On Again?
summary
Argues that a simultaneous reboot can leave a site unrecoverable because of hidden runtime dependencies and bootstrapping cycles, shows dependency graphs (generated with Graphviz) that reveal surprising edges, and describes an internal gRPC access-control change that monitors then blocks unexpected RPCs to prevent new dependencies. Recommends automated restore drills and treating fallbacks like packed reserve parachutes, keeping fallback plans tiny and robotic, enforcing layered bootstrapping and using network policies (e.g., Kubernetes NetworkPolicy) to make dependencies explicit and testable.
AI-written summary · about