Building a Data Science Platform with Argo Workflows
summary
Describes a working data‑science control plane that uses Argo Workflows plus a small agent per customer cluster to schedule across many clusters, a hosted Postgres API as source‑of‑truth, and per‑cluster message queues (SQS today) for pull‑based scheduling. Features include a CLI, HTTP API, git‑tag-driven workflow template versioning, RBAC, and support for customer logging backends. Demo shows submitting DAGs to remote and local clusters, viewing live logs and pod state — it works end‑to‑end.
AI-written summary · about
chapters (6)
- 0:00Start
- 4:29Common Requirements
- 13:32Bloomberg Architecture: Deployment API
- 15:19Bloomberg Architecture: Scheduling API
- 16:34Pipekit Architecture
- 19:20Contrasts and Comparisons
related talks
20mDemystifying Argo Workflows: An Architectural Deep Dive
25mEvolving Argo Workflows for AI: New Features, Tuning, and Multi-Tenant Isolation
29mLarge Scale Batch Processing with Argo Work...
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...
22mLeveraging Argo WorkflowTemplates Within a Data Platform
27mLeveraging Argo Workflows To Move, Migrate and Manage All of Your Workflows
25mData Science Workflows Made Easy: Python-Powered Argo for Your Organization