Scaling Data Platforms: Managing Growth from Zero to Billions of Transactions
summary
Arcesium scales a data platform that processes ~120 million records/day and ~70k pods/day with 5k pods/hr variance by using Argo workflows for event-triggered processing, S3+SQS+Kafka ingress, EKS, and switching to Cilium CNI to remove IP allocation bottlenecks. Engineering changes include raising sensor/controller QPS and bursts, per-client quotas, exponential backoff to avoid thundering-herd, image caching, and DataDog observability; result: cluster spins from ~2.5k to ~9–10k pods and cuts a 1–1.5 hour window significantly.
AI-written summary · about
chapters (12)
- 0:00Introduction to scaling
- 2:31Data processing flow overview
- 5:20Challenges of dynamic workloads
- 8:09Common scaling misconceptions
- 9:57Cluster scaling bottlenecks
- 13:10Observability and simulation
- 13:53First choke point sensor pod
- 16:50Second choke point controller pod
- 18:15Kubernetes API throttling
- 20:02Thundering herd problem
- 22:08Results after optimizations
- 22:50Key takeaways and summary
related talks
47mScaling: from 0 to 25 million users
08mPlatform Engineering at Scale
28mPrepare Your System To Scale OR Why Auto Scaling Is Not Enough
22mScaling Argo Events for Enterprise Scheduling: A Case Study fr...
41mBuilding a platform to process £34bn in card transactions
27mScalable by Design: How to Build an Elastic Cloud Infrastructure...
31mTaming 50 Billion Time Series: Operating Global-Scale Prometheus Deployments on Kubernetes
44mIntroduction to application scaling