Chaos Engineering for Cloud Foundry Platform
summary
T‑Mobile runs Cloud Foundry at scale (13+ foundations, ~39,000 containers, ~700M daily transactions, 3,000+ apps) and builds chaos tooling to prove resilience: turbulence++ (Go) adds VM/process kills, CPU/memory stress, packet loss/latency and SSH pauses; monarch (Python) discovers where an app runs and injects service‑level faults (DB latency, blocked ingress/egress, instance crashes) to validate circuit‑breakers and run Game Days. Demos show attacks succeed and auto‑recovery works; current limits: one cluster targeted at a time and tool consolidation/Envoy sidecar plans are underway.
AI-written summary · about
chapters (15)
- 0:00Introduction and Alaska Airlines story
- 4:12Cloud Foundry platform at T-Mobile
- 8:45Problem statement and chaos engineering
- 11:16King and falcon story
- 13:19T-Mobile's chaos engineering journey
- 16:30Infrastructure level chaos with Turbulence++
- 22:46Live demo: Infrastructure chaos
- 31:46Application level chaos with Monarch
- 38:05Live demo: Application chaos
- 40:36Graduation path to production chaos
- 41:18Fortune UI application setup
- 51:29Game days and future plans
- 52:53Chaos as a ladder metaphor
- 53:35Wrap-up and keynote preview
- 54:24Q&A session
related talks
57mChaos Engineering Primer
27mCloud Foundry in Production
38mCloudy with a Chance of Chaos
33mBeyond the Monkeys Chaos Engineering on the Cloud
19mChaos Engineering at Scale
40mChaos Engineering at Scale
39mMake Cloud Native Chaos Engineering Easier - Deep Dive into Chaos Mesh
29mBoF Deep Dive: Chaos Engineering