An exploration of black holes: strange failure modes
summary
Demo code runs on Google Cloud Shell with miniKube and Gremlin, deploying a Bank of Anthos app to experiment with black hole attack on real services. Blackholing the balance reader makes the UI show balance reader shows --- while deposits succeed but payments fail with connection refused. Scaling fixes appear: a single pod becomes unavailable, so issuing kubectl to scale replicas to two lets transaction history failover; blackholing 50% of pods still preserves history. Region tests run as 60s blackholes; automation via Gremlin API, Jenkins or Drone is recommended.
AI-written summary · about
chapters (17)
- 0:00Start
- 1:05An Exploration on Black Holes Strange Failure Modes
- 4:01What Is a Black Hole
- 5:32How Do We Create a Black Hole
- 8:00Duplicate Transactions
- 13:21Can I Deposit Funds and Send Payments
- 15:38Clone the Repo
- 16:19How a Black Hole Impacts Our Transaction History Service
- 21:25Additional Replicas
- 22:08The Different Sizes of Black Holes
- 23:58How Do Black Holes Impact Observability
- 24:31What Can We Learn from Black Hole Related Failures
- 27:19Networking Failure Modes
- 29:39What Tools and Test Frameworks To Automate Chaos Engineering Would You Recommend
- 30:09Adding It to Your Ci Cd Pipeline
- 31:07Chaos Engineering Api
- 32:20Where Shall I Start
related talks
29mKubernetes Failure Stories and How to Crash Your Clusters
34mChaos in the cloud
34mChaos Debugging: Finding and Fixing Microservice Weak Points
33mLive Hacking Cloud Architectures - Thorsten Jakoby and Mirna Alaisami
40mJourney into the World of Chaos Engineering
35mKubernetes, Lessons Learnt
42mPanel - Klustered: eBPF Edition
36m101 Ways to Crash Your Cluster