Running open large language models in production with Ollama and serverless GPUs
summary
Shows deployment of Gemma 2 (9B) quantized to ~5GB running via Ollama + llama.cpp on Cloud Run with Nvidia L4 (24GB VRAM), using LangChain, LangGraph and Gradio; container cold-start to GPU ~5s, model load ~15s, demo limits max instances 3 and per-instance concurrency 4, scales to zero and works in production. Demonstrates code-generation with CodeGemma, sandboxed execution, routing between “answer” and “write+execute code” paths, and practical safeguards for sandboxing and outbound request limits.
AI-written summary · about
chapters (16)
- 0:00Introduction and demo
- 2:20Gemma model overview
- 5:55Ollama inference server
- 8:47GPU requirements
- 10:58Building the application
- 13:24Benefits of open models
- 14:26Cloud Run deployment
- 17:14Scaling and authentication
- 19:19Concurrency limits
- 22:08Agentic systems
- 25:21CodeGemma demonstration
- 29:41Query routing challenges
- 31:02Code execution workflow
- 32:07Sandboxing untrusted code
- 34:42LangGraph orchestration
- 36:54Q&A and conclusion
related talks
27mYes You Can Run LLMs on Kubernetes
12mDemo: Building cloud-native, AI-powered applications with GKE
05mBuild your own LLM on Cloud
12mDemo: Rapid prototyping with Gemma and Llama.cpp
08mDemo: Gemma on-device with MediaPipe and TensorFlow Lite
01h 15mWorkshop: Working with Gemma and Open LLMs on Google Kubernetes...
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine