Managing Artifacts at Scale for CI and Data Processing
Talkat ArgoCon Europe 2023 · on Tuesday, 2023-04-18· at 11:25· Elicium Building | Elicium Ballroom 2 [Argo Workflows]·
summary
Covers concrete Argo features and configs: use Key-only artifacts via ConfigMaps to avoid repeating S3/MinIO settings, embed workflow UID in artifact keys to prevent collisions, and use pod spec patch to allocate more CPU/memory for large artifact processing. Demonstrates a 20-way fan-out/reduce job and a CI build+deploy to Argo CD using MinIO. Describes artifact garbage collection (Argo 3.4) that runs deletion pods in the user namespace for S3/GCS/Azure and lets artifacts be deleted on completion or workflow deletion.
AI-written summary · about
chapters (10)
- 0:00Introduction to artifacts
- 1:25Defining artifact types
- 2:50Storage options
- 4:21Data passing strategies
- 5:55Centralizing configurations
- 8:09Naming and organization
- 10:20Scaling resources
- 13:18Artifact garbage collection
- 17:14Live demonstrations
- 23:37Q&A and wrap up
related talks
39mBuild and Deploy Your Application with Argo on Kubernetes
25mHow to Multiply the Power of Argo Projects By Using Them Together
34mInfrastructure at Scale Using ArgoCD
23mGitOps Meets AIOps: Building Reliable AI Pipelines With Argo
18mUnveil the Secret Ingredients for Argo CD in the Enterprise-Scale
35mHow To Build Production Grade DevOps Platform Using Argoproj
43mGitOps at Scale with and Argo CD on
29mLarge Scale Batch Processing with Argo Work...