Building a machine learning team and platform at Cash App
summary
Cash App integrates Amazon SageMaker into an existing Kubernetes-based service called Gondola to fix slow p99 hosting and lack of observability, packaging models with a metadata file (gondola.json) and offering single model endpoints or cost-saving multi-model endpoints grouped by team or use case. A reverse-proxy process called Homogenator translates requests to Triton, shared-memory optimization is planned, Datadog dashboards monitor traces, and the platform routinely handles about 2–3k QPS at peak.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:26Background
- 2:22Existing model hosting
- 3:05What to invest in
- 3:45The journey
- 5:48ML to the product
- 7:37Gondola
- 14:10Model deployment
- 17:17Model inference
- 24:08Monitoring
- 28:43Scaling
related talks
19mDrive transformation through machine learning
34mLat Designing and Building Complex Machine Learning....
42mThe Challenges Managing a Kubernetes-Based Machine Learning Infrastructure
20mIndustrial-Scale Machine Learning
37mStreamlining Machine Learning Operations
29mMachine learning in production
03mInterview
26mManaging ML Models @ Scale