Designing a Hybrid AI Platform on Kubernetes
summary
Edge deployment on k3s clusters uses k3s + Rancher to centralize GitOps delivery while GPU inference runs on Nvidia servers with up to 128 GB unified memory and ~1,000 TOPS per node connected by ConnectX for low‑latency multi‑GPU inference. The platform uses dify and LangGraph for low‑code or flexible agent orchestration, stores models on NFS separate from container images, automates updates via GitOps, and already runs a 120B parameter model designed to scale toward 400B parameters.
AI-written summary · about
chapters (13)
- 0:00Hybrid AI platform intro
- 1:29Cloud AI challenges
- 5:52On-premise AI challenges
- 7:41OneBox AI architecture
- 9:09AI agent deployment details
- 12:50Query processing flow
- 14:18GPU server clustering
- 18:29GitOps management portal
- 20:14App deployment workflow
- 23:08App update automation
- 26:50Maintenance engineering use case
- 28:15Hybrid AI benefits
- 29:18Conclusion & Q&A
related talks
25mEmpowering AI-Driven Drug Discovery: Overcoming Challenges in Building a ML Platform on Kubernetes
06mAI Inference at the Edge: Beyond the Hype
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
27mHarnessing Kubernetes for Scalable AI/ML Workloads
26mBuilding a Cloud Native Edge Platform for Accelerating AI Inference
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
06mEdge AI in Practice: Lessons from Producti...