Building Reliable Infrastructure for LLM Workloads on K8s
summary
Claims a production-ready K8s pattern: use a BGP anycast VIP for API server failover, peer each node with two top‑of‑rack switches and run bird on the host for routable pod CIDRs, use BFD to shorten failover, and manage GPUs with the Nvidia operator plus device plugins. Recommends anti‑affinity, taints/tolerations, spare GPU capacity, and 90/10 canary rollouts; shows this combination yields predictable scheduling, visible network paths, and faster fault diagnosis under real LLM load.
AI-written summary · about
chapters (12)
- 0:00Platform Reliability Matters
- 0:49Why LLMs Stress Infra
- 1:39From Experiment to Production
- 2:23Reference Architecture Overview
- 3:26Failure Domains and Capacity
- 4:51Control Plane Anycast VIP
- 5:56Routable BGP Networking
- 8:00GPU Node Preparation
- 9:54Scheduling for Resilience
- 11:16Safe Rollouts with Gateway
- 12:07Observability and Checklist
- 12:53Closing Thoughts
related talks
32mBeyond Round-Robin: GPU-Aware Load Balancing for LLM Inference in Kubernetes
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes
24mReliability Is Not Just About Balancing Loads
20mInferencing LLMs in production with Kubernetes and KubeFlow
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
16mKeynote: LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
30mSecuring LLMs in Kubernetes: Best Practices