Fast and Furious: Practice in Horizon Robotics on Large-scale End-to-end Model Training
Talkat KubeCon + CloudNativeCon China 2025 · on Tuesday, 2025-06-10· at 13:45· Level 19 | Crystal Court I·
summary
Horizon Robotics and Alibaba Cloud describe how they run end-to-end large-model training for autonomous driving on Kubernetes, focusing on training Sparse4D-like perception models across a 2000-GPU RDMA cluster. They explain practical solutions for handling massive video data and many small files (distributed data caching), making scheduling decisions with network-topology awareness and job affinity, detecting and diagnosing failures, and restoring interrupted jobs via backup-machine replacement. The talk also shares operational experience using CNCF projects Volcano, Fluid, and NPD.
AI-written summary · about
related talks
46mMachine Learning with Kubernetes
34mDeep Learning at Scale
36mMulti-Cloud Machine Learning Data and Workflow with Kubernetes
38mDemocratizing Deep Learning at Scale with Horovod
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...
28mBuild, Buy and the Golden Ratio: How Theator scaled up its continuous training framework
18mAccelerating 3D simulation workflows with NVIDIA Omniverse and DGX Cloud on Azure
17mDevOps for Private AI: A Scalable Platform Using Kubeflow