Serverless for ML Inference on Kubernetes: Panacea or Folly?
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Thursday, 2020-11-19· at 17:40· Intrado Virtual Event Platform,·
summary
Benchmarks compare AWS Lambda, Google Cloud Run (Knative) and Verta containers on Kubernetes (ModelDB/KubeFlow workflows) for ML inference and find concrete tradeoffs: warm-start p50–p99 latencies show Verta ~2x faster, cold starts on serverless take tens of seconds versus <1s because Verta keeps one replica warm, Cloud Run reaches steady 100 QPS in ~33s and Lambda is about twice slower, many models (DistilBERT ~256MB, BERT ~2×) or 20GB embedding indexes exceed serverless limits (memory>3GB, CPU>4), and serverless lowers TCO but can raise pure infra cost for steady workloads.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:40What is Serverless
- 9:40ML Serving Considerations
- 11:20Benchmark
- 16:51Usability
- 24:19Cost
- 25:29Summary
related talks
18mServerless AI's Rise: Revolutionizing ML Deployment
37mFrom Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes
41mContainers or Serverless Functions - When to choose what?
41mIs Serverless the End of Kubernetes?
46mServerless for Data Scientists
39mBuilding model testing infrastructure and scaling AI predictions
01h 00mServerless Madness on Kubernetes
34mBuilding scalable end-to-end deep learning pipelines in the cloud