Bug Hunting in Distributed Systems: Using Robustness Tests to Test...
summary
etcd builds a robustness-test framework that injects live failures with gofail, runs varied traffic profiles and mixed-version clusters, and validates histories with Porcupine to catch consistency breaks. the suite runs on every PR, reproduces historical bugs (e.g., issue 14370 where revision jumps backward from 165 to 164 and watches travel back in time), exposed crashes under high load and stale pod-status reports, and prevents regressions by failing CI until fixes restore linearizability.
AI-written summary · about
chapters (14)
- 0:00Distributed system testing intro
- 3:17Traditional tests limitations
- 5:09Robustness testing approach
- 7:17Data inconsistency case study
- 9:47Jepsen limitations
- 11:11Failure types in systems
- 12:14Robustness test architecture
- 15:45Designing test scenarios
- 16:27Validation with Porcupine
- 18:53State machines for validation
- 23:07Robustness testing workflow
- 23:52Live robustness test demo
- 27:04Key takeaways
- 29:08Q&A
related talks
39mTesting Level Hard: Distributed Apps
27mAutomating Resilience Testing with Clojure and Docker
31mTesting Kubernetes CRDs
25mHow to ensure Testing Robustness in Continuous Testing Environments
34mBreaking the Build Before It Breaks You: The Magic of E2E Tests
20mScalable and Accurate Application-Level Crash-Consistency Testing(…)
40mResilience Testing - Failure is an Option
55mThe art of living with broken things