Breaking the 1.5MB Barrier: Running Large Metaflow Flows with Argo for AI/ML Workloa...
summary
Metaflow flows convert to Argo workflows on EKS but hit the etcd 1.5MB limit when node status grows, causing runtime "request entity too large" failures after hours of wasted compute. The solution configures offloading node status to PostgreSQL (Aurora RDS), wired into the Argo controller, and enables 10,000-node Argo workflows in production instead of the ~3,000 default; implementation runs across regions and needs IAM auth and fair‑share scheduling next.
AI-written summary · about
chapters (11)
- 0:00Breaking the 1.5MB barrier
- 1:03What is Metaflow
- 4:17AI/ML vs infrastructure
- 8:57Metaflow adoption
- 11:27Large Argo workflows
- 13:565MB etcd limit
- 15:02Offloading to Postgres
- 16:49Decoupling controllers
- 17:53Scaling to 10k nodes
- 18:16Future work and security
- 19:17Q&A
related talks
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...
07mArgo in an AI-Native World: Effective In-Cluster Machine Learning and A...
35mBuilding Robust Reactive ML Systems In A Multiplayer Setting
40mOptimizing AI/ML Workloads
25mPanel: Argo for ML: Achieving Scalability and User Experience
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
22mStreamlining Big Data Workflows with Memoization & Work Avoidance