8 Million Tasks Daily: Driving Efficiency in Multi-Tenant Data Platforms at Hori...
recording removed on YouTubehttps://youtu.be/F3dWgIquCQk
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Describes how Horizon Robotics rebuilt a multi-tenant data platform for autonomous-driving workloads using Kubernetes tooling to run 8 million tasks per day. It covers replacing per-workflow agents with a centralized Argo Workflows Agent Service (an “agentless” design) to cut costs and node count, refactoring metadata and reducing VolcanoJob sizes to eliminate an Informer bottleneck and cut sync latency from ~4 minutes to under 10 seconds, and using Kueue and Volcano for elastic quotas and resolved preemption to improve stability in large-scale multi-tenant environments.
AI-written summary · about
related talks
25mOne Engine To Rule Them All: Unifying Cloud Workloads With Argo Workflows
29mBuilding a Batch Processing Platform...
02mThe Problems We're Here to Solve
27mThe Life of an Argo Workflows Salesperson
24mDynamic Big Data in the Cloud with Kubernetes
29mLarge Scale Batch Processing with Argo Work...
33mData: The Land DevOps Forgot
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions