Running Massively Parallel Deep-learning Inference Pipelines on Kub... Suneeta Mall & Martin Abeleda
summary
Nearmap builds a custom inference engine called Seer CRD on Kubernetes that uses gang scheduling and per-workflow RWX volumes to run massively parallel, stateful DAGs. The system exhausts K80 spot GPUs, runs thousands of nodes and tens of thousands of pods, and produced semantic content over 1,000,000+ km² at 5cm/pixel in two weeks (about 70,000 km²/day). Operational fixes include a DB-backed queue, multi-region clusters, autoscaler fallbacks, and abandoning unsafe GPU sharing for code and bidding limits.
AI-written summary · about
chapters (11)
- 0:00Nearmap overview
- 1:06AI at Nearmap
- 3:20End-to-end AI workflow
- 6:07Building the Seer solution
- 9:01Cloud infrastructure
- 10:49Scaling and scheduling
- 12:58Storage and networking
- 15:37Cluster management
- 17:18Multi-region deployment
- 19:16Challenges and misadventures
- 22:19Future work and Q&A
related talks
30mMaster large-scale ML batch inferencing
30mRealizing End to End Reproducible Machine Learning on Kubernetes
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
41mPeer Beneath the Surface of the Earth
28mProductionizing Machine Learning Models at Scale with Kubernetes
29mA Scalable Platform for Training and Inference Using Kubeflow
34mDeep Learning at Scale
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow