Scaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
summary
PepsiCo claims production deployments of 70B-parameter Llama models on Kubernetes, hosting them internally for data control while running demos on A100 node pools with MIG 20GB slices. The platform uses an NVIDIA stack — NVIDIA GPU Operator, Triton inference, DCGM metrics, Istio, Grafana, persistent volumes and a GitOps pipeline to pull model weights, run bench tests and enforce CR approvals. They report workable low-latency inference after tuning, but warn GPU scarcity, strict governance and the need for GPU-aware orchestration.
AI-written summary · about
chapters (10)
- 0:00Introduction and team intro
- 3:11Agenda and company overview
- 7:47AI adoption and enterprise impact
- 10:15PepsiCo’s AI use cases
- 13:22Technical platform stack
- 17:40Deployment strategy and governance
- 20:07Demo of LLM deployment
- 29:07Key learnings and challenges
- 32:19Future roadmap and priorities
- 33:21Q&A and closing remarks
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
27mHarnessing Kubernetes for Scalable AI/ML Workloads
31mScaling AI for Industry
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
22mGenAI Training In Production: Software, Hardware & Network Considerations
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
28mNavigating LLM deployment: Tips, tricks, and techniques