Building an ML Application Platform from the Ground Up
summary
Claims that BentoML lets teams separate runtimes for IO and inference so preprocessing runs on memory‑heavy nodes and model inference runs on GPU or CPU nodes, with runners that scale independently. Shows a credit application example using a PyTorch fraud model plus three scikit learn risk models run in parallel, and a runtime feature of adaptive batching that improves throughput. Builds deployable Bento container images and a Kubernetes CRD with autoscaling and Grafana observability, and reports this pipeline runs in production.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:30Model serving problems
- 3:55Off-the-shelf limitations
- 6:03DIY solution challenges
- 8:29ML serving requirements
- 11:17BentoML framework
- 12:02Credit app with BentoML
- 14:39BentoML deployment blocks
- 17:27Container deployment
- 18:30Microservice deployment
- 20:17Model serving comparison
- 21:00Summary and BentoML vision
- 21:22Q&A
related talks
54mRipped Machine Learning with Distributed Computing
28mBuilding Developer-Centric ML Inference Platforms
40mBuilding highload ML powered service
33mBuilding Resilient (ML) Pipelines for MLOps
29mBuilding an ML Platform from Scratch
30mAccelerating AI Application: NAVER's Journey from Model to Application
04mInterview
41mInfrastructure for ML Applications