One Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
summary
Argonative claim: Argo Workflows can act as a single scheduling engine by either migrating workloads into Argo or integrating existing systems via proxy steps. Implementation shows ~12 clusters, the largest with 5k cores, and workflows that once hit 30k pods. Patterns include a Slurm proxy that sents jobs and waits, Airflow→Hera one-day migrations reusing Python functions, and a UV template for ad hoc Python functions. Works well for batch analytics but struggles with low-latency streaming, specialized hardware, and cross-cluster federation.
AI-written summary · about
chapters (16)
- 0:00Argo Workflows intro
- 1:05Problem and Argo solution
- 2:32University’s Argo journey
- 7:15Diverse workflow engines
- 9:40Can Argo unify workloads?
- 10:21Migration vs integration
- 12:11Migration decision matrix
- 13:37Migrating HPC workloads
- 15:22Integration with existing systems
- 17:08Migrating Airflow workflows
- 18:34Spark and serverless integration
- 19:58Unified dev experience
- 21:02Argo limitations
- 22:26Benefits of templates
- 23:07Argo future wish list
- 24:54Q&A
related talks
29mLarge Scale Batch Processing with Argo Work...
20mDemystifying Argo Workflows: An Architectural Deep Dive
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
06mComparing Argo Workflows and Apache Airflow at Scale
36mArgo Workflows: Intro, Update and Deep Dive
22mCross Cluster Execution of Argo Workflows
27mThe Life of an Argo Workflows Salesperson
27mLeveraging Argo Workflows To Move, Migrate and Manage All of Your Workflows