Leader Election: Pitfalls and Alternatives
summary
Leader election requires explicit fencing not just consensus: a lost leader can keep acting if downstream systems accept stale writes. Google Cloud Router’s control plane (the VRM/VRC pair) suffers a blocking init bug that lets old leaders stay connected, triggering mass deprogramming and an outage lasting 8 hours 34 minutes. The recommended fixes are fencing tokens (fencing token (sequencer)) and leader-aware proxies; long-term mitigations include increased timeouts, HA sharding and moving to dynamic sharding or databases instead.
AI-written summary · about
chapters (8)
- 0:00Leader election dangers
- 1:01What is leader election
- 5:13Why leader election isn't enough
- 10:03Fencing in distributed systems
- 15:20Outage from fencing flaws
- 19:33Operational challenges
- 24:22Alternatives for state management
- 27:55Conclusion
related talks
23mLightning Talks on Distributed Systems
26mUnlocking Diego: Achieving Distributed Robustness Through Simplicity and Reliability
33mWhy Are Distributed Systems So Hard?
59mSolving Raft's practical problems in Tarantool. What, how and why
41mThe Raft Consensus Algorithm
47mDistributed Systems: When Democracy and Computer Science Come Together
46mWhat can go wrong in a distributed system - experience from...
22mWhy Are Distributed Systems so Hard?