FiDe: Reliable and Fast Crash Failure Detection to Boost Datacenter Coordination
summary
FiDe reports crashes in datacenters in under 30 μs (7.2× faster than prior art) by creating stable end-to-end interactions: FiDe reports crashes <30 μs (7.2× faster). It uses a kernel Kprobe OS watchdog and packet interception via XDP/EBPF hooks to detect and broadcast process removal, enforces quota-based traffic engineering with IP multicast and dual redundant trees to bound network latency, and ships a controller for join/rate assignment. Experiments log 200+ billion packets, estimate a critical dual-tree failure once every ~23 years, and show up to 2.23× throughput and 0.46× latency when integrated into a key-value store and a coordination service.
AI-written summary · about
chapters (12)
- 0:00FiDe introduction
- 1:02Failure detectors in systems
- 2:52Gray failure limits
- 4:13FiDe’s sync approach
- 5:41Two-domain architecture
- 6:23OS watchdog in FiDe
- 7:05Network reliability in FiDe
- 8:11Redundancy for network failures
- 9:15FiDe performance results
- 10:21FiDe for coordination
- 12:30Simpler protocols with FiDe
- 13:54Conclusion
related talks
26mF10: A Fault-Tolerant Engineered Network
32mActive Fault Finding in Networks
12mDebugging Transient Faults in Data Centers using Synchronized Network-wide Packet
01h 12mRedundancy does not imply fault tolerance
25mEnsuring Connectivity via Data Plane Mechanisms
25mResilience for AI Workloads at Scale: The Fast and the Finicky!
23mBalancing on the Edge: Transport Affinity without Network State
16mPRED: Performance-oriented Random Early Detection for Consistently Stable Performance...