Testing the Resilience of a SEAPATH Cluster: In Search of Single Points of Failure
summary
A three-node SEAPATH cluster is exercised with 36 fault scenarios — about 35 virtual‑machine tests and 11 container tests — injecting failures at cluster membership, storage, virtual networking, hypervisor/runtime, OS/kernel, PTP and power levels. The cluster uses Corosync and Pacemaker with shared storage but no fencing/STO NITH, and observed recoveries are typically 1–5 seconds with occasional ~30s cases; no VM loss is seen. Results recommend enabling fencing, tuning Corosync election strategy, adding cluster and PTP monitoring, and automating the test suite.
AI-written summary · about
chapters (13)
- 0:00Introduction and context
- 1:09Virtualization in substations
- 3:03Reliability comparison with conventional systems
- 6:24CPATH platform overview
- 8:53High availability cluster setup
- 12:08Resilience testing methodology
- 15:48Virtual machine test results
- 17:31Corosync and Ceph resilience findings
- 20:04Container resilience testing
- 22:34Container test results
- 23:49Conclusions and recommendations
- 26:23Future work and automation plans
- 27:12Q&A session
related talks
22mEnsuring Requirements on the SEAPATH Project: Testing Process and Continuous Integration
32mSEAPATH: What's New - Recent Improvements and Future Developments
26mWhat Do Virtualization and SEAPATH Really Change for RTE ?
46mChaos Experiments - Datacenter Stress Testing
16mEngineering Failure-Resilient Systems
42mHow to stay alive even when others go down:...
42mPlanning To Fail
23mA Case Study on Testing LFE Seapath at National Grid Electricity Transmission