Soft Failures, Hard Goals - Accelerating Payments at Scale During the Pandemic
summary
A 90-day resilience project timeboxes six workstreams—alerts and metrics, reconciliation, test automation, anti-fragile design, education and automation—and produces measurable results: payment engine moves $96B/year, test suites reach 75% automation coverage using unit tests, Mockito and GitLab, RCAs complete within three business days, nonblocking async queues remove performance leeches, and an objective of 152 consecutive incident-free days is met, cutting customer-impact incidents dramatically and eliminating many overnight pages.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:34Initial Impact of COVID-19 Pandemic
- 2:23Payments Batch Processing
- 3:19Challenges
- 4:45Why focus on batch processing?
- 7:05Leaders Practice Extreme Ownership
- 10:49Resilience as a project
- 14:15Alerts & Metrics
- 18:23Antifragile Design - Assume Failure
- 20:48Accomplishments
- 26:34Operational Readiness
- 29:05Business Continuity Plan (BCP)
- 31:45Vendor Management
related talks
07mScaling Financial Inclusion: High-Performance Payment Systems
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation
33mThe Road to Enable DevOps Within the Payment Industry
18mAutomating a Payments Engine
55mThe Troubles of Automating “all the Things”
01h 02mFive things every developer should know about building mission-critical systems
01h 03mMoving Fast - without crashing
10mBeyond “try harder”: Effective strategies to tackle bugs