Scaling to Thousands of Data & CI/CD Pipelines Using Argo and Virtual...
summary
Pipekit provisions thousands of Kubernetes tenants so customers can run Argo Workflows almost immediately, using Argo Workflows to trigger a workflow that creates a virtual cluster and bootstraps tooling. Virtual clusters run as a pod with their own API server and syncer (vcluster), start in about 39 seconds on average, and cost roughly $90/year each versus ~$900/year for an EKS control plane, yielding ~85% cost savings and roughly 2x throughput in a hello‑world scaling test (2k → 5k runs). Tools: Argo CD, image caching, renovate, and GitOps for lifecycle.
AI-written summary · about
chapters (8)
- 0:00Scaling with Argo and V Cluster
- 4:42Virtual clusters for Kubernetes
- 12:19Demo: V Cluster with Argo
- 15:09Trial cluster setup
- 17:17Argo workflows for provisioning
- 19:49Performance gains with V Cluster
- 21:34How V Cluster works
- 25:26Q&A
related talks
26mGitOps Me Some of That! Managing Hundreds of Clusters with Argo CD
26mDog Food Delight: How Argo Workflows Eats Its Own CI
19mGitOps: Unlocking the Power of Kubernetes Cluster Management
29mLarge Scale Batch Processing with Argo Work...
06mArgo CD and Kubernetes from event based to microservices
23mArgo at Scale: Navigating Complex Multi-Dimensional Deployments Across H...
25mHow to Multiply the Power of Argo Projects By Using Them Together
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows