Team Teardown: Driving Platform Adoption and Success
summary
Spotify describes a production ML stack built around kubeflow (SpotifyQFlow) plus a shared cluster and CLI, plus serving and feature products named Salem, Jukebox, ML Home; the platform team is ~30 people and an engagements function onboarded 169 people across 42 teams in 2020 and delivered 4 end-to-end ML solutions. Engineers use TFX components, Luigi, and a component SDK to write reusable custom components (80/20 design), teams contribute back to the platform, adoption speeds deployment but edge cases and prioritization still require hands-on engineering.
AI-written summary · about
chapters (7)
- 0:00Introduction to Team Teardown
- 1:48ML at Spotify overview
- 7:25Role of engagement team
- 14:48Supporting varied maturity levels
- 20:43Scope of ML platform
- 24:56Big challenges in platform management
- 29:31Closing and resources
related talks
38mBuilding and Managing a Centralized Kubeflow Platform at Spotify
40mPowering Spotify's Audio Personalization Platform
32mTeam Teardown: Evolving an ML Platform
32mScaling Kubeflow for Multi-tenancy at Spotify
42mThe Challenges Managing a Kubernetes-Based Machine Learning Infrastructure
32mTeam Teardown: Organizing and Supporting ML
27mEmpowering Traceable & Auditable ML in Production
32mKubeflow Deep Dive