Panel: AI Infra Best Practices: Enterprise Do’s and Don’ts
summary
Kubernetes hosts separate clusters for batch, training, and inference at Uber, with Ray for training and Nvidia Triton for serving; GPU capacity remains scarce so clusters often map to fault domains and cloud providers, complicating DR and workload consolidation. Moving from baked images toward CUDA/ROCm sidecars and exposing vendor metrics via an API (migrating to DCGM metrics) improves portability. Early wins: disaggregating CPU pre/postprocessing from GPU inference raises GPU utilization; RDMA and multicluster Ray enable emerging inter‑GPU distributed training.
AI-written summary · about
chapters (15)
- 0:00AI infra best practices intro
- 1:06Uber’s AI products & infra
- 5:29Kubernetes for AI workloads
- 7:37Cluster segregation AI vs non-AI
- 9:46Multi-cloud GPU strategy
- 12:12Cluster management & scheduling
- 14:18Disaster recovery GPU infra
- 17:12Distributed training on Kubernetes
- 18:56Cluster size for training & inference
- 21:06Enterprise AI readiness
- 23:34Observability & logging challenges
- 26:25Vendor ecosystem advice
- 28:57Key takeaways for AI infra
- 30:39Future of multi-cluster AI
- 32:48Q&A
related talks
16mAI + Kubernetes: What Beginners...
40mPanel: Kubernetes and GPU Trends for AI in Financial Services
15mKeynote: From Platforms to AI Factories. Has Kubernetes Solved It?
30mDesigning a Hybrid AI Platform on Kubernetes
24mImproving GPU Utilization and Accelerating Model Training with Scheduling Framework and NRI
40mAI Training & Inference on Kubernetes: Compute Patterns and Best Practices with Karpenter.
27mHarnessing Kubernetes for Scalable AI/ML Workloads
36mProduction GPU Cluster with K8s for AI and DL Workloads