Growing Pain: Scaling to 10K Workflows per Week
summary
Bloomberg runs a centralized multi‑tenant Argo-based platform that reaches 10K workflows per week and hits real limits: repeated API timeouts from an API server overload and wildly variable tenant SLAs. Response work includes building dashboards and forwarding logs/metrics to an internal observability platform, creating a custom reconciler controller that emits per‑namespace metrics and auto‑remediates stuck pods, and using cluster sharding plus a unified API to rebalance tenants and limit blast radius.
AI-written summary · about
chapters (9)
- 0:00Scaling workflow orchestration
- 2:49Tenant cluster isolation
- 3:30Data center resiliency
- 3:50Observability needs
- 6:45Scaling workflow platforms
- 7:06Custom metrics implementation
- 12:30Multicluster federation
- 14:16Tenant workload isolation
- 18:39Q&A
related talks
23mArgo at Scale: Navigating Complex Multi-Dimensional Deployments Across H...
24mScaling Argo CD: From Symptoms To Solution
26mGoing with the (work)flow? Incident Response for Vicious Workflows
31mEscaping operations hell
40mManaging to Your SLO Amidst Chaos
28mPrepare Your System To Scale OR Why Auto Scaling Is Not Enough
24mTaming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...
37mAutomatically Detect the Top Performance & Scalability Issues in Distributed