Children's Guide To LLMs on Kubernetes
summary
Shows vLLM running on a three‑node Kubernetes cluster with a CPU vLLM pod and a Docker image (~2–3 GB) exposing a FastAPI endpoint on port 8000; demo queries hit a GPT‑2 container and return answers. Claims vLLM fixes the two production bottlenecks—exponential KV cache growth and GPU underutilization—by paged attention (blockwise memory) and continuous batching so GPUs fill with large parallel batches. Also covers DRA/DRNet and NUMA-aware placement; demo works but uses CPU so answers are lower‑quality.
AI-written summary · about
chapters (17)
- 0:00Generative AI and LLMs intro
- 3:15Generative AI and multimodal models
- 4:21Transformer architecture basics
- 5:44Why Kubernetes for AI
- 6:27Dynamic resource allocation
- 7:33DRNet for AI networking
- 8:39NUMA awareness for LLMs
- 9:41LLM inference challenges
- 12:32Introducing VLMs
- 13:58VLMs optimize GPU usage
- 14:41Kubernetes and VLM integration
- 16:05VLM system features
- 17:53VLM architecture deep dive
- 20:03Demo setup and cluster
- 21:32Python and Docker for VLM
- 25:27Running LLM inference demo
- 27:12Future of VLMs and contributions
related talks
42mStop Treating LLMs Like REST APIs
45mEverything You Need to Know About Running LLMs Locally
25mLLM Powered Agents with Kubernetes
47mGive Your LLMs a Left Brain
34mImprove AI Inference (serving models) With KServe and VLLM
21mScaling LLM Inference with vLLM
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM