From Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes
summary
Claims a real deployment path that cuts latencies from seconds to under 100 ms and boosts throughput about 10x by running models serverless on Kubernetes (Cloud Run / Knative / KEDA / KNative discussed). Demos run a 60s 400 users benchmark across VM, Kubernetes, serverless CPU and GPU; Flask/WSGI → FastAPI/ASGI async code halves overhead, GPU shows little gain for serial workloads. Covers IaC for infra, prefetching images, scale-to-zero, quantization (Triton/TensorRT), ONNX+TVM and dynamic batching.
AI-written summary · about
chapters (8)
- 0:00AI inference scaling challenges
- 1:22Why AI projects fail
- 8:48Inference dilemma and CAP theorem
- 13:23VMs vs Kubernetes vs serverless
- 22:13Live demo: inference performance
- 28:13Key takeaways and optimizations
- 33:31Avoiding cloud lock-in
- 36:19Q&A
related talks
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
27mServerless for ML Inference on Kubernetes: Panacea or Folly?
39mDelivering Serverless Experience on Kubernetes: Beyond Web Applications
36mAccelerating High-Performance Machine Learning at Scale i...
36mAccelerating Machine Learning at Scale with
27mHarnessing Kubernetes for Scalable AI/ML Workloads
15mAI Model Inference on K8s
29mServerless on Kubernetes: What We Learned From Customers