Empowering Traceable & Auditable ML in Production
summary
Spotify runs a centralized ML platform called Hendrix that is built for traceability and auditability and whose core claim is a unifying Hendrix Ontology plus a metadata backbone that records where pipelines run, data sources, and experiment outputs. Hendrix exposes a managed multi-tenant KFlow (TFX-based) golden path with canonical components for production and now generalizes automated experiment tracking to ad hoc Ray workloads so teams can log experiments consistently. This reduces friction, increases reuse across serving and pipelines, and enables future model inventories and automated model cards.
AI-written summary · about
chapters (7)
- 0:00Hendrix ML platform intro
- 1:07What is Hendrix at Spotify
- 3:54Hendrix ML ontology
- 6:11Metadata as backbone
- 8:36Experiment tracking case study
- 15:38Hendrix future roadmap
- 17:03Q&A with Jonathan
related talks
29mHow Spotify Does ML At Scale
40mPowering Spotify's Audio Personalization Platform
30mTeam Teardown: Driving Platform Adoption and Success
26mML Product Experiments at Scale
38mBuilding and Managing a Centralized Kubeflow Platform at Spotify
33mCompliant Data Management and Machine Learning on Kubernetes
01h 21mBuilding a ML Model Observability pipeline
10mML-based Log Analysis for Faster Debugging