Surviving Crypto Volatility: How to Build Resilient Infrastructure
summary
Uphold runs on Cloudflare fronting EKS with Aurora and ElastiCache and a single production cluster of ~100 namespaces, 265 services, 1,000 deployments, 3,000 pods on 66 nodes (~1,400 CPUs, 3 TB RAM). A surge from ~3M to 10–15M requests lifts P95 from ~100 ms to 2–3 s and drives DB CPU to 70–80%. Fixes: HPA tuning, switching NodeJS API pods to one core per pod, isolating critical services in a separate node pool, adding distributed tracing, batching/data loaders and Redis pipelining. The next spike shows stable latency (~200 ms) and low DB waits.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker
- 1:20Uphold platform overview
- 6:24Crypto volatility impact
- 9:16Bitcoin spike incident analysis
- 12:11Post-incident infrastructure upgrades
- 14:43Contention and system bottlenecks
- 16:57Application code optimizations
- 21:40Next spike resilience test
- 24:51Resilience philosophy and conclusion
related talks
26mGrowing Pain: Scaling to 10K Workflows per Week
14mKeynote: The Big Leap: Conquering the Legacy System and Building a Scalable Cloud Infra...
27mScalable by Design: How to Build an Elastic Cloud Infrastructure...
42mScaling Resilient Systems: A Journey into Slack's Database Service
15mScaling Observability Stack for AIOPS
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
36m101 Ways to Crash Your Cluster
31mDevOps in the Cloud: Case Studies of Amazon.com teams and their resilient architectures