The playbook for a sovereign Model-as-a-Service platform
summary
A concrete blueprint for running a private MaaS on Kubernetes that uses OpenShift AI + LLMd + OCI/Quay to pull weights, expose OpenAI-compatible endpoints and manage deployments as Kubernetes CRDs. Demo deploys a 4B coin model onto shared GPUs with vLLM/Ollama for inference, Grafana + DCGM for GPU and token observability, and per-user tiers that chargeback tokens. The platform enforces 50-request rate limits (50 requests succeed, 51th is blocked), autoscaling replicas, and keeps data on-prem/hybrid.
AI-written summary · about
chapters (9)
- 0:00Sovereign Model-as-a-Service intro
- 3:05Coffee and AI models analogy
- 9:46Shadow AI and external APIs problems
- 12:34Optimal Model-as-a-Service approach
- 14:17Deploying model on OpenShift AI
- 16:28Kubernetes custom resources
- 18:42Testing and managing model tiers
- 21:51Platform monitoring and observability
- 24:18Conclusion and benefits
related talks
42mDefense against the dark arts of AI: The playbook for a sovereign model-as-a-service platform
35mCase Study: AI-as-a-Service on Kubernetes at Scale and In Production
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
36mShift-Left for LLMs: Securing the AI Model Supply Chain
24mDemo of Kubernetes at the Edge for AI Model Deployments
28mNavigating LLM deployment: Tips, tricks, and techniques
34mServing ML Models at Scale with Seldon and Kubeflow