Empowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
summary
Migrates from Airflow to a multi‑cluster Argo Workflows deployment that manages 3,000+ GPUs and schedules over 1M tasks per day, using Argo CD/GitOps for identical cluster rollout. Implements namespace‑level concurrency, tiered quotas, a cache-vs-informer reconciliation to avoid duplicate creates, offloads heavy event work to user-side handlers, and integrates Ray for GPU job packing. System sustains ~20,000 concurrent workflows, achieves 11× faster runs and about 30% cost reduction, and upstreams fixes to Argo.
AI-written summary · about
chapters (12)
- 0:00BYD's million-task journey
- 1:26Automatic annotation challenges
- 3:14Moved from Airflow to Argo
- 5:46Scaling with multi-cluster Argo
- 7:19GPU and CPU optimization
- 8:52Ray clusters for GPU workloads
- 13:36Concurrency and stability at scale
- 17:14Argo stability improvements
- 21:50Monitoring large-scale workflows
- 23:57Argo fixes contribution
- 24:40Key learnings and results
- 26:07Q&A
related talks
29mLarge Scale Batch Processing with Argo Work...
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
25mEvolving Argo Workflows for AI: New Features, Tuning, and Multi-Tenant Isolation
23mArgo at Scale: Navigating Complex Multi-Dimensional Deployments Across H...
25mPanel: Argo for ML: Achieving Scalability and User Experience
20mDemystifying Argo Workflows: An Architectural Deep Dive
27mBuilding a Data Science Platform with Argo Workflows
26mLeveraging Argo Ecosystem for Scalable Cloud-Native Solutions at Databricks