Workshop: Working with Gemma and Open LLMs on Google Kubernetes...
summary
Hands‑on workshop that provisions a RAG app via Terraform on Google Kubernetes Engine, fine‑tunes and serves Gemma fine-tunable models pulled from Hugging Face (gated token required) using the TGI inference server, and creates embeddings with Ray for embedding stored in Cloud SQL using PGVector in Cloud SQL. Deployment targets GKE Autopilot with L4 GPUs or spot nodes, Terraform takes about 30 minutes, a Jupyter notebook ingests PDFs, a simple chat frontend runs on Gemma 2B and demonstrates retrieval and moderation working end to end.
AI-written summary · about
chapters (11)
- 0:00Workshop introduction
- 1:04Google Cloud setup
- 14:10Hugging Face token access
- 21:18Terraform for RAG
- 30:48Introduction to Google Kubernetes Engine Autopilot
- 42:07Ray for ML workloads
- 52:18Gemma and open models
- 59:46RAG deep dive
- 1:07:14Demo prep
- 1:08:43Live RAG demo
- 1:14:10Q&A
related talks
12mDemo: Building cloud-native, AI-powered applications with GKE
27mYes You Can Run LLMs on Kubernetes
05mBuild your own LLM on Cloud
09mUsing and fine-tuning Gemma 3
44mBuilding amazing Front-End Project with Gemma: A State-of-the-Art Open Model
10mDemo: Using Gemma with the Hugging Face ecosystem
43mRunning open large language models in production with Ollama and serverless GPUs
09mGetting started with Gemma models