Executing Chaos Engineering in Production at a Critical Financial Institution
summary
A financial cloud platform running Kubernetes, Redis, Kafka and external APIs adopts a reproducible chaos platform with governance guardrails, safety windows, automated rollback and integrated observability, turning experiments into self‑service validation. Results include 73% reduction in MTTD, 10 hidden vulnerabilities exposed and 5 new metrics; GameDays across five teams and 10 scenarios surface cascade failures (session drop during 50s health‑check timeouts, DNS and circuit‑breaker surprises) and force design fixes rather than knob‑turning.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:21Why chaos engineering matters
- 3:15Cloud migration resilience
- 4:42Repeatable resilience framework
- 6:12Structured chaos scenarios
- 8:23Scaling chaos platform
- 11:03Tooling for production chaos
- 12:18Self-service learning capability
- 14:08Realistic game days
- 15:58Resilience team shift
- 17:03Case: Active-active cluster failure
- 19:20Case: Network failures simulation
- 21:17Predictable recovery lessons
- 23:10Outcomes from chaos engineering
- 23:31Adoption challenges
- 25:01Key takeaways
- 26:25Closing remarks
related talks
25mChaos Engineering Applied To the FinTech Domain
54mControlled Chaos: The Inevitable Marriage of DevOps & Security
01h 01mEmbrace chaos to achieve stability
29mMaking Sense of Chaos: Implementing Chaos Engineering in a Fintech...
30mRisks in Systems Design: Chaos Engineering in Apps & Cloud Security
40mNavigating the Unknowable: Resilience through Security Chaos Engineering
40mSecurity & Chaos Engineering: Resilient Distributed Systems
43mCyber Chaos Engineering