Unlock the Full Potential of Generative AI via Microservices and Istio Service Mesh
summary
A Python Streamlit demo runs local Llama models (Alama with Nama 3.2) and OpenAI on a single-node Kubernetes cluster and shows GenAI microservices enrolled into Istio ambient mode with Istio CNI and a ZT trust tunnel. Calls between services use mutual TLS, metrics appear in Cyale, and Waypoint plus a service entry and authorization policy enforce egress control (OpenAI restricted to POST). The demo swaps models dynamically, observes TCP/Layer4 metrics, and requires no app code changes to enable mesh features.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:25Generative AI excitement
- 3:14AI system components
- 6:07Demo setup with Istio ambient
- 9:19LLM demo with observability
- 15:48AI children’s book cover analysis
- 21:15Mood detection from photo
- 24:50LLM response comparison
- 26:36Istio Waypoint egress control
- 29:45Demo recap
- 30:29Q&A
related talks
42mEffortlessly Build, Run, Secure, and Manage Traffic for a Generative AI Application From...
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
43mLLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale
08m⚡ Lightning Talk: Unleash the Power of Generative AI Using Only Open Source Technolgies
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
32mOrchestrating AI Models in Kubernetes: Deploying Ollama as a Nati...
01h 01mBuilding Generative AI Applications in Go- Gari Singh, Google (Part 1)
01h 42mBuilding Responsible AI Agents in Decentralized Open-Source Environments