How Form3’s real-time payment platform survives cloud outages
summary
Claim: platform survives loss of an entire cloud provider with no customer impact. Architecture runs active/active/active across AWS GCP and Azure with a Kubernetes cluster per cloud, a single logical CockroachDB cluster (9 nodes requiring a 5-node RAFT quorum) and a global NATS JetStream layer. Engineering adds a custom cross-cluster pod disruption budget to avoid rolling-restart quorum loss. Disaster recovery is validated in production by intentionally shutting down a cloud for 24 hours; an actual GCP outage previously causes zero payment failures.
AI-written summary · about
chapters (11)
- 0:00Regulator’s cloud exit mandate
- 1:11Technology failure intolerance
- 2:35Original single-cloud architecture
- 4:41Surviving cloud outages
- 6:46Multi-cloud with Kubernetes and CockroachDB
- 9:55Why CockroachDB for resilience
- 12:27Operational challenges of own stack
- 13:51Database resilience across clouds
- 16:37Disaster recovery testing
- 18:26Proof the platform works
- 19:51Lessons from multi-cloud system
related talks
40mBut can you really run your app on 2 clouds at the same time?
28mGlobal Payments: Subsecond Performance Worldwide
53mResiliency and Availability Design Patterns for the Cloud
54mDistributed Caching in Real-Time Payment Systems
31mFuture-proofing operational data at scale
42mLeveraging Multi-Cloud Clusters in Real-World Scenarios
50mWeathering the Cloud Storm: Building Resilient Geo-Distributed Apps
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation